A clever way to transfer bulk data via PeopleSoft Integration Broker with Segmented Messages

 Introduction:

    We all know that the data transfer between systems is present since the day we invent Internet of Things, and we invented many mediums and methodologies to do that. So as PeopleSoft IB for integrating another PeopleSoft Application or 3rd Party application either to send or receive or for both. And I would say that we haven't explored all the possible ways of data transfer or integration between two systems especially on PeopleSoft IB. Recently I had a chance to go through a situation where the data (I mean a huge volume of data like files and media) needs to send and receive between third-party application and PeopleSoft application and without loosing the track of which one to process and in which order. In the search of the answer when weighing the PeopleBook I came across two broadly used ways 1. Message Segmentation 2. Binary Data transfer. In this blog we will deep dive into the first approach.

Fundamentals:

    The approach here is to split the message into multiple segments and send/receive and track at the opposite site using transaction ID. Following things you have to ensure before writing any code to send or receive the message and processing the same at either end.

1. Your PeopleSoft application needs to be configured as Segment Aware at the NODE level (where it integrates with third-party).
2. Setting up the maximum no. of segments that the application can or should handle.
3. Creating and managing the message segments. like splitting the payload as multiple junks. Either by content length or by it's grouping. (like you can send Invoice details in 1st segment and in the 2nd segment you can attach the scanned copy and in 3rd segment Invoice summary)
4. Carefully handling the segment number when sending and receiving messages
5. Deciding where to store segmented messages until you receive all the segments. You can either choose either Memory or Application Table (your admin can help here)
6. Choose whether your segments are send or received in Ordered or Unordered
7. Understanding API header parameters which will tell the PeopleSoft application which segment been sent and how much is remaining.

How it's done (Step by Step)

Node Configuration:

Setup maximum no. of segments to handle:

Creating and managing messages in Segments:

    When working with messages either JSON or XML, you decide the split based on parts or content. I'll just give a simple example for your understanding.

Part 1 of 3

<?xml version="1.0"?>

<root>

    <segment>1</segment>

    <invoiceheader>

        <descr>....</descr>

    </invoiceheader>

</root>

Part 2 of 3

<?xml version="1.0"?>

<root>

    <segment>2</segment>

    <invoicelines>

        <line>

            ...

        </line>

        <line>...

        </line>

    </invoicelines>

</root>

Part 3 of 3

<?xml version="1.0"?>

<root>

    <segment>3</segment>

    <invoicesummary>

        <invoicetotal> .. </invoicetotal>

            ...

    </invoicesummary>

</root>

When you combine the messages on your code the final message looks like below, 

<?xml version="1.0"?>

<root>

    <invoiceheader>

        <descr>....</descr>

    </invoiceheader>

    <invoicesummary>

        <invoicetotal> .. </invoicetotal>

            ...

    </invoicesummary>

    <invoicesummary>

        <invoicetotal> .. </invoicetotal>

            ...

    </invoicesummary>

</root>

Handling Segment Number in Sending or Receiving message:

    The message will be send from Integration Broker PubSub Handler to the actual handler code, only if all the messages (the number given on ChunkCount) are received. So, while processing the message you can able to read through all the segments and decide on the process. For this example I have just combined the message and populated in to a log record.

Segment Ordering:

    When creating message queue ensure to select whether it is Ordered/Unordered processing. When you have bulk messages flow through system it's better to have Unordered and let your application package/handler code deal with the combining and splitting the messages.

Understanding API Parameters:

• DataChunkCount: Indicates the total number of data chunks or message segments contained in the transaction.
• DataChunk: Indicates the number of the data chunk or message segment that you are sending.

Apart from these two there are many other header parameters to include to properly designate the call to appropriate service operation. Most of these parameters are self-explanatory, so I'm not going to detail it here.

Full Demo:

All the project related items (Project Backup, Codes) are shared in my GitHub Channel for you to download and explore.

Hope this helps in your career or at least gives you some knowledge.

Happy Learning.:)

{"IsRecording":0,"IsBrowserHooked":0,"IsPaused":0,"IsResumed":0,"IsFirstRec":0,"IsStopped":0,"recCurrState":"NewRecOpened"}