Abinitio Training: Medium Complex Components

Abinitio Training
Medium Complex Components

> Data Aggregation
0345Smith
0345Smith Bristol
Bristol 56
56 Bristol
Bristol 63
63
0212Spade
0212Spade London
London 88 Compton
Compton 12
12
0322Jones
0322Jones Compton
Compton 12
12 London
London 31
31
0492West
0492West London
London 23
23 New
New York
York 42
42
0121Forth
0121Forth Bristol
Bristol 77
0221Black
0221Black New
New York
York 42
42
Confidential ©2012 Syntel, Inc.

> Data Aggregation of Sorted/Grouped Input
0345Smith
0345Smith Bristol
Bristol 56
56
0121Forth
0121Forth Bristol
Bristol 77 Bristol
Bristol 63
63
0322Jones
0322Jones Compton
Compton 12
12 Compton
Compton 12
12
0212Spade
0212Spade London
London 88
0492West
0492West London
London 23
23 London
London 31
31
0221Black
0221Black New
New York
York 42
42 New
New York
York 42
42

> Built-in Functions for Rollup
 The following aggregation functions are predefined and are only

available in the rollup component:
avg max
count min
first product
last sum

> Rollup Wizard
Note the use of an aggregation function in the expression

ROLL UP
 Rollup evaluates a group of input records that have the

same key, and then generates records that either
summarize each group or select certain information from
each group.


Two ways to use ROLLUP
 You can use a Rollup component in two ways,

depending on how you define the transform
parameter:
 Define a transform that uses a template rollup
function. This is called template mode and is
most often used when you want to aggregate the
data.
 Create a transform using an expanded Rollup
package. This is called expanded mode and
allows for rollups that do not necessarily use
regular aggregation functions.

Template Mode
 In the transform parameter, you specify an aggregation

function that describes how the data should be “rolled up,”
or summarized, usually in some cumulative way
 For example ,count and aggregate.

Expanded Mode
 Expanded mode provides more control over the

data rollup
 specify transformations that are not possible with
template mode
 With an expanded Rollup package, you must
define the following items:
 DML type named temporary_type
 initialize function that returns a temporary_type
record
 rollup function that takes two input arguments
(an input record and a temporary_type record)
and returns an updated temporary_type record
 finalize function that returns an output record

Package View of a Rollup

Rolloup Transform

Vector Creation Through Rollup

SCAN
 For every input record, Scan generates an output record

that consists of a running cumulative summary for the
group to which the input record belongs, up to and
including the current record
 For example, the output records might include successive
year-to-date totals for groups of records.

Template Mode
 Template mode is the simplest way to use SCAN. In the

transform parameter, you specify an aggregation function
that describes how the cumulative summary should be
computed.

Expanded Mode
 Expanded mode provides more control over the

scan. It lets you edit the expanded package, so
you can specify transformations that are not
possible with template mode
 With an expanded SCAN package, you must
define the following items:
 DML type named temporary_type
 Initialize function that returns a temporary_type
record
 Scan function that takes two input arguments (an
input record and a temporary_type record) and
returns an updated temporary_type record
 Finalize function that returns an output record

NORMALIZE
 Normalize generates multiple output records from each of

its input records
 You can directly specify the number of output records for
each input record
OR
 You can make the number of output records dependent on
a calculation

Run Time Behavior of Normalize
 Reads the input record.

 Performs temporary initialization
 Performs iterations of the normalize transform function
 Sends the output record to the out port

Simple Example
 An Input file having vector data is normalized

Simple Example
 Output is as below:

Phase
 A phase is a stage of a graph that runs to completion

before the start of the next phase
 By dividing a graph into phases, you can make the best use
of resources such as memory, disk space, and CPU cycles
 The boundary between two phases is called a phase break,
and it belongs to the first of the two phases

CheckPoint
 Checkpoint is a point at which the Co>Operating

System saves all the information it would need to
restore a job to its state at that point
 You can have checkpoints only at phase breaks
 As the execution of the graph successfully
passes each succeeding checkpoint, the
Co>Operating System:
 Deletes the information it has saved to be able to
restore the job to its state at the preceding
checkpoint
 Deletes the temporary files it has written in the
layouts of the components in all phases since the
preceding checkpoint
 Commits the effects on the file system of all
phases since the preceding checkpoint

Pictorial Representation
 Please see below:

CHECK ORDER
 Check Order tests input records to determine

whether the records are sorted according to the
key specify in the key parameter
 Limit :Maximum number of incorrectly ordered
records Allowed

COMPARE CHECKSUMS
 Compare Checksums compares two checksums

generated by the Compute Checksum component
 You can have checkpoints only at phase breaks
 As the execution of the graph successfully
passes each succeeding checkpoint, the
Co>Operating System:
 Deletes the information it has saved to be able to
restore the job to its state at the preceding
checkpoint
 Deletes the temporary files it has written in the
layouts of the components in all phases since the
preceding checkpoint
 Commits the effects on the file system of all
phases since the preceding checkpoint
COMPARE RECORDS
 Compare Records reads records from two flows,

compares the records one by one, and writes
one-line text reports
 It compares records using a byte-to-byte
comparison
 Limit

COMPUTE CHECKSUM
 Compute Checksum computes a checksum for

records
 Output record format
record
big endian real(8) sum_of_crc;
big endian real(8) sum_of_lengths;
unsigned big endian integer(4) xor_of_crc;
unsigned big endian integer(4) record_count;
end

GENERATE RANDOM BYTES
 Generate Random Bytes generates a specified

number of records, each consisting of a specified
number of random bytes

VALIDATE RECORDS
 Validate Records separates valid records from

invalid records

Questions ?

Abinitio Training: Medium Complex Components

Uploaded by

Copyright:

Available Formats

Abinitio Training: Medium Complex Components

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Abinitio Training: Medium Complex Components

Uploaded by

Copyright:

Available Formats

Abinitio Training

Medium Complex Components

Confidential ©2012 Syntel, Inc.

Confidential ©2012 Syntel, Inc.

 The following aggregation functions are predefined and are only

Confidential ©2012 Syntel, Inc.

Note the use of an aggregation function in the expression

Confidential ©2012 Syntel, Inc.

 Rollup evaluates a group of input records that have the

Confidential ©2012 Syntel, Inc.

 You can use a Rollup component in two ways,

Confidential ©2012 Syntel, Inc.

 In the transform parameter, you specify an aggregation

Confidential ©2012 Syntel, Inc.

 Expanded mode provides more control over the

 finalize function that returns an output record

Confidential ©2012 Syntel, Inc.

Confidential ©2012 Syntel, Inc.

Confidential ©2012 Syntel, Inc.

Confidential ©2012 Syntel, Inc.

 For every input record, Scan generates an output record

Confidential ©2012 Syntel, Inc.

 Template mode is the simplest way to use SCAN. In the

Confidential ©2012 Syntel, Inc.

 Expanded mode provides more control over the

Confidential ©2012 Syntel, Inc.

 Normalize generates multiple output records from each of

Confidential ©2012 Syntel, Inc.

 Reads the input record.

Confidential ©2012 Syntel, Inc.

 An Input file having vector data is normalized

Confidential ©2012 Syntel, Inc.

Confidential ©2012 Syntel, Inc.

 A phase is a stage of a graph that runs to completion

Confidential ©2012 Syntel, Inc.

 Checkpoint is a point at which the Co>Operating

Confidential ©2012 Syntel, Inc.

 Please see below:

Confidential ©2012 Syntel, Inc.

 Check Order tests input records to determine

Confidential ©2012 Syntel, Inc.

 Compare Checksums compares two checksums

 Compare Records reads records from two flows,

Confidential ©2012 Syntel, Inc.

 Compute Checksum computes a checksum for

Confidential ©2012 Syntel, Inc.

 Generate Random Bytes generates a specified

Confidential ©2012 Syntel, Inc.

 Validate Records separates valid records from

Confidential ©2012 Syntel, Inc.

Confidential ©2012 Syntel, Inc.

You might also like