RedTitan Enterprise Data Processor: Reference semantics (version 1.40)

Apply ApplyFDD Assert AskWizard Attr AttrData Buffer Capture Clean Concat Copy Count CSVparse DataAttr DataBase Date Def Demote Dispose Edit EncodeCSV EncodeXML Eval Execute External

FDDdata FDDparse FileName Files FileWatch Format Group HostInfo HTML Install Internal Join LookUp LookUpS MakeDir Match Merge Modify MsDOS OpenWizard Pages Print Promote

Readfile Rename Revise SaveWizard Select Separate Sort SQL Substitute Summary Sync Table TaskList TCP TCPserver Text Timer Unwrap View Wrap WriteFile XMLparse Appendix

Execute

This function runs a program each time a target element is encountered. The command line is constructed by means of a script contained in the content of the options. Option attributes specify the program name, input target and whether processing will wait until the program terminates execution.

Example:

Port	Value
DATA (in)	<FILE NAME="c:\ndp\ref\rivers.csv"/>
DATA (out)

MsDOS

This function runs an input stream as an MsDOS BAT script and awaits its completion. The standard output is collected and made available as the LOG output stream.

Example:

Print

This function makes a Windows Shell-services PRINT request on a document.

HTML

This function displays the data stream that flows to/through it in a window for the user to examine but which is assumed to be XHTML data. It appears directly in the nQ remote access display and so permits an arbitary HTML display to be available for remote browsing.
Note that the window will not appear until some actual data has been read. An empty data stream does not cause the window to be created.

Table

This function displays data attributes in tabular form, both in a local window and also in the remote browser display. Each element at the targetted level generates one table row.

Edit

A diagnostic function that displays data as it flows through the block, but with the possibility that the user can edit it before it is output.

View

This function displays the data stream that flows to/through it in a window for the user to examine. Useful as a diagnostic aid. Note that the window will not appear until some actual data has been read. An empty data stream does not cause the window to be created.

AskWizard

This function displays a wizard panel to ask the user for an input value. The main text area of this dialogue is filled in from the data content of the properties after substituting values from the attributes of the current input element (if any). The output element is modified by adding an attribute whose value is the filename specified. The original content is retained.

A dialogue box is displayed to the user when a level 0 element is received, or at startup if there is no input source. When the user enters a response and answers OK then the result is output along with its associated content. If the user answer CANCEL then no output is generated (and the input element is lost).

Example1:

OpenWizard

This function displays the file-open wizard panel to the user - and awaits a filename specification. The main text area of this dialogue is filled in from the data content of the properties after substituting values from the attributes of the current input element (if any). The output element is modified by adding an attribute whose value is the filename specified. The original content is retained.

A dialogue box is displayed to the user when a level 0 element is received, or at startup if there is no input source. When the user chooses an existing filename and answers OK then the result is output along with its associated content. If the user answer CANCEL then no output is generated (and the input element is lost).

Example1:

SaveWizard

External

This function converts a data stream to a compact textual representation suitable for short-term storage in external media or transmission across a communications channel. Note that such a data stream can only definitely be read back by the same version of the program and so should not be archived. This function is used by the spooling and database functions. The data stream can optionally be encrypted for security in transmission.

Example1:

Example2:

Internal

This function converts the compact textual representation created by External back to the original data stream. This function is used by the spooling and database functions.

Example1:

Example2:

CSVparse

Read elements of raw data in CSV format and convert to a data stream. The raw data content of each element is converted to a list of CSV elements, each of which contains a list of elements (named FLD1, FLD2 etc.) with data content being the CSV fields.

Example:

EncodeCSV

Write a data stream as CSV data. The data values are taken first from the attributes, and then from the element content.

Example:

EncodeXML

Write a data stream as XML data.

Example:

XMLparse

Parses the raw data stream as XML data. This parser supports UTF-8, ISO-8859-1 and ASCII encodings at present - and also UCS-2 and UCS-4 but without auto-recognition. Internally, the parser is an SGML parser and is slightly forgiving of some of the more strict syntax rules in XML. It will also parse a sequence of multiple documents sourced from a single data stream.

Example:

FDDparse

This function reads an FDD data description (as created by the RedTitan Page Designer) and converts it to a data stream suitable for use by ApplyFDD.

Example:

ApplyFDD

This function decodes raw data using an FDD specification, which means that it processes row+column data descriptions and also tagged data. The FDD specification would normally be generated by FDDparse

Property	Default	Description
TARGET	1	This can either be a level number (0 being the outermost level) or the name of the elements to be selected. The targeted elements supply raw text data from which fields are identified and output.
PRESERVE	false	If "true" is specified then the original data is retained. The extracted fields are appended to the raw data.
FIXEDWIDTH	variable	If an integer value is specified then lines are broken after the fixed number of characters specified. A zero value requests that the fixed line length is calculated from the data specification. By default lines are terminated by a line-end marker.
FIXEDHEIGHT	variable	If an integer value is specified then a record is complete after the specified number of lines have been read. A zero value requests that the line count be calculated from the data specification. By default the record terminates at the end of the data content.

FDDdata

This function constructs row/column data from attribute values in the input data stream according to an FDD specification. The FDD specification would normally be generated by FDDparse

Match

This function cuts up data content by searching for specified strings of characters.

• A matched region starts when the string specified by BEGIN is encountered. If not specified then the region starts with the first data character.
• The matched region finishes when the string specified by END is encountered or when the target level ends. If not specified then the BEGIN string is used and the region finishes before the next BEGIN is encountered.
• If NEST is true then an inner region can be started by a further instance of the BEGIN string.
  If false then the region must finish before another start is considered.
• Where both the BEGIN and END matches at the same position in the data then the longest match will be used. If BEGIN and END are the same string then the region both starts and finishes at the same place. The effect of this depends on the state of NEST, where false results in the match string alternately starting and ending a region, and true results in only the match string itself becoming the region.
• The DISPOSITION setting now deterines what happens to the match string itself. The default action is to not include it in the output. Other possibilities are to have it included either inside or outside the matched region thereby making it part of the outlying data or as part of a region. The special (0) option puts it outside either of these, and in the special case of the BEGIN and END strings being the same in the nested mode then the match string appears as the regions only content.
• Data that is not included in any region can be optionally packaged as specified by the DATA option, which includes the possibility of having this outlying data removed from the output.
• The matched region is packaged according to the TAG specification. This allows for it becoming the content of a new element, or being prefixed and suffixed by specified character data.

Text

This cuts up data into lines by looking for [RET] and/or [LF] characters and replacing them with the EOLN marker. It will optionally package the lines into pages either delimited by a FORMFEED ($0C) character occurring anywhere in the data, by a specified TAG character occurring at the start if a line, and by a line count being exceeded.

Example:

Pages

This makes the specified target elements into ASCII data pages by removing the element header and replacing the end tag with a FormFeed ($0C) character. It is effectively the inverse of the "Text" function when the "PAGE" option had been specified.

Timer

This function generates an element at repeated intervals, providing a convenient way of prompting a routine process.

Example:

DataBase

This function provides information about ODBC databases. It will list the available ODBC database names, the table names within a database, or column names within a table.
The function can be used with out without an input stream.
Note that ODBC database access requires that your system has the 32-bit ODBC access mechanism installed and configured appropriately and also that your nDP system has the nDPODBC.DLL installed.

SQL

This function makes an SQL query to the specified database and reads the results. By default a query is constructed to read the whole of the specified table.

Note that ODBC database access requires that your system has the 32-bit ODBC access mechanism installed and configured appropriately and also that your nDP system has the nDPODBC.DLL installed.

LookUp

This function uses one stream to construct a database and then performs relational lookups on it merging the resulting data into the key stream.

The database stream (upper) is read to construct a database for the start of each new bacth on key records. If the stream terminates then the last loaded database is reused. A new batch of key records is deemed to start whenever a containing element of the last used key ends. This means that you can use the same database to process many files of data records, or have each file of keys looked up in a corresponding database.

Example:

LookUpS

This function perfoms the same database lookup function as the LookUp function, but has the extra feature of recording statistics about the database use which become available when the database is closed.

The database stream (upper) is read to construct a database for the start of each new bacth on key records. If the stream terminates then the last loaded database is reused. A new batch of key records is deemed to start whenever a containing element of the last used key ends, unless the ONCE option is specified as true. This means that you can use the same database to process many files of data records, or have each file of keys looked up in a corresponding database.

The first output stream is exacly the same as for the LookUp function, being a copy of the second input stream but potentially with extra data merged from the selected database record.
The second output stream provides the usage statistics. It is a copy of the database stream (albeit with any duplications removed) in key value order with an additional attribute added to each record element giving the number of times that the record was transmitted to the output stream.

Note: if you do not want the record access information then use LookUp as recording the usage information makes this function slower.

MakeDir

This function creates file directories to receive files. The directory is extracted from the specified filename and will be built if it does not already exist. Use this function immediately before writing a file or renaming a file to ensure that the destination directory exists.

ReadFile

Reads one or more disc files.

Example1:

Example2:

WriteFile

Writes data to one or more disc files.

Example:

Rename

Renames or deletes files.

Example:

FileWatch

Watches for files that match the specified name and details, renames them to a 'work in progress' filename, and finally renames them or flags them as archived when required to supply the next one. If a file is found that cannot be opened for update or is of zero length then it will be ignored (as it is assumed that it is still being created).
There are two different modes of operation:

• If no input stream is used then a continuous (never ending) search is made for files that match the WATCH specification. Files are then checked for full exclusive access before an output element is generated.
• If there is an input stream then the input elements supply filenames which (after applying the WATCH specification) are individually awaited. Only when the file exists, and is accessible will any output be generated and the next input element read.

Files

Generates a list of filenames from a wild-card specification.

Example:

FileName

This function adds a new attribute value to element headers by applying wildcard filename rules to an attribute that names a file.

Example:

Install

Install tasks using a description encoded as a data stream.

TaskList

List a task's composition. The result could be used to install another copy.

Concat

Concatenates two entire data streams.

Example:

Merge

This concatenates corresponding pairs of elements from up to 3 data streams.

Example1:

Example2:

Join

This function joins up to three data streams on a first-come first-served basis. Each input is polled and the first element to arrive will be copied to the output. If more than one input stream is active simultaneously then elements are taken from each stream in turn.

Example:

DataAttr

This folds element data content back into the enclosing element's header making them into attribute definitions.

Example:

AttrData

This copies attribute values into the element content, moving the attribute value to the content of an element with the same name. Non-text attribute values are converted to the standard text representaion.

Example:

Substitute

This uses attribute values from the main data stream to fill in parameters specified by elements in two other streams (often set up as constants). One of these determines the output attributes and the other determines the content.

Example:

Attr

This function creates a new attribute for a element, the value being made by the substitution of a script.

Example:

Capture

This function captures data following the target element and appends it to the target element content.

Example:

Promote

This function copies attribute values from the first element of a level to the parent element header.

Demote

This function copies attribute values from target elements to specified inner levels.

Clean

This function deletes attributes from element headers.

Dispose

This function deletes attributes from element headers.
It can also remove specified content from within the target element.

Wrap

This function wraps an entire data stream with a header, and thus will convert a list of data objects into a single object containing the list.

Note that this function can be used inside the APPLY function. In that situation, each APPLY substream is individually wrapped - but make sure that the wrapper source stream repeats so as to keep supplying fresh wrappers.

Example:

Unwrap

This function removes the outer element headers from a data stream.

Example:

Summary

This function summarises a group or potential group of elements.

Group

This function adds a wrapper around groups of elements. There are a number of ways that a sequence of elements can be identified to be made into a group.

• One or more attirbutes can be identified as having to be of the same value within the group, i.e. a group terminates before an element with different attribute values is encountered. When this grouping method is active the wrapper elements inherit the attributes that are the same within the group.
• A specific test can identify an element to start a new group.
• A specific test can identify an element to be the last in the group.
• A minimum and maximum group size can be specified to group elements by number.

Copy

This duplicates a data stream so that it can be used as input to two functions.

Example:

Buffer

This function encourages pipelining by slightly reading ahead of its output. Normally, data processing occurs on demand - but there can be situations where there is no reason why a function could not start processing its next input before the demand has become apparent (e.g. because of a hold-up waiting for the completion of an external process).

Example:

Count

This function counts elements and inserts an attribute into the element headers supplying their number in the stream. Since the increment can be obtained from the input stream it can also be used to accumulate a total.

Note: this function is currently only defined for integer values and ascending counts.

Example1:

Example2:

Eval

This function allows string values to be converted to numeric form (e.g. for numeric-order sorting) and can sum a number of fields.

Sort

This function sorts a list of elements in a data stream.

Example1:

Example2:

Separate

This function creates a new data-stream containing the elements that are common to both input data-streams. A record is taken from each input stream and their values compared. If they are the same, the element is recorded (once) in the new data-stream (EQUAL) and removed from both output streams (DATA1, DATA2). Only the residue of the (unequal) records remain in their respective output data-streams.

Revise

This function changes the names of attributes and tags. The description of how to change the names can either be explicitly supplied, or derived from the first targetted element (useful for CSV files which name their fields in the first record).

Example:

Format

This converts attribute values to text and formats them, with the possibility of adding or removing leading and/or trailing spaces.

Date

This function will perform simple date calculations using dates in the ISO representation.
Where FROM and USING both specify dates, the result is the difference, in the specified units (as a real numeric value). Where FROM is not specified it defaults to the current date and time, and so the result is the age of the date specified by USING in the specified units (default seconds).
Where a date and a numeric value are specified, the result is a date made by subtracting the numeric value (scaled by the UNITS to seconds) from the date.

Example:

Select

This compares element attributes, selecting those that pass the specified test. The rejects are available as an outer-level data stream.

Example:

Assert

This tests element attributes, selecting those that pass the specified test. If a test fails then the task is halted with an error.

Sync

This function synchonises the flow of data of two streams. The second stream is held up until the required target element passes through the first.

Apply

This function selects elements from the main data stream and supplies them, or just their contents for separate processing. The results are then recombined with the main stream.

Note that because of the way this function works it may not be possible to use it with any arbitary set of functions to process the inner stream. In the current implementation the inner stream is continuous, and is delimited by special processing instructions. If the inner stream is structurally altered so as to loose or duplicate these special markers then the APPLY function will not operate correctly and may not terminate or give a "Data structuring error". You must not make use of the markers as the details of the inner stream delimiting is subject to change.

Modify

This function primarily permits arbitary modifications of element headers by means of an ECMAScript program. There is also some provision for simple modification of the element content.

The data content of the properties is compiled by the ECMAScript compiler and the result stored as global variables. This permits the definition of functions that will be used to do the processing. The current element is denoted by the global symbol "$".

Def

This function provides a way to associate names with values in the tasks local symbol table. The current main use for these symbol values is to parameterise task specifications as far as filestore directory names is concerned. The symbol names can be used at the start of a filename specification followed by double colon - and this notation is replaced by the associated value. Where no value is known the functional block that is accessing the symbol name will stop and wait until it is defined.
You can use this functional block quite unconnected to the rest of the task activity to create a particular named symbol at task startup.
Note that task local symbols cannot have there values changed once they are defined although no error will be raised if an attempt is made to redefine the local symbol with the same value.

HostInfo

This network service obtains information about a specified host from the domain name server. A host can be specified by name or IP address. The information returned depends on the domain services.

Example1:

TCP

This network function implements a simple TCP connection.
An attempt is made to make a connection to the specified port on the specified host. Note that the host can be specified by IP address or host name, and the port can be specified by number or service name.
If successful, the data content of the target element is transmitted to the specified port, then an end-of-data condition is signalled and data is received from the host and copied to the main output until the host signals end-of-data.
When the connection was unsuccessful, the target element (only - no outer levels) is copied to the reject output.

TCPserver

This network function implements a simple TCP server. The server listens on the specified port - and performs a simple data transfer to clients.

+ + + + APPENDIX + + + +

Substitution

Conditional expressions

Global symbols

Back to Contents

Copyright RedTitan Technology Limited 1999-2005