Friday, January 3, 2025

Universal Data Loader

    After a decade of time, I'm back to blogging, And it's time to give back the community that helped me grow in my career and skills with some of works.

    As a first step, let me present my efforts on the vision to create a common upload utility I wanted since I started as a developer. Many customers have their own modules (bolt-on or customized) developed on their application and using delivered systems only when necessary.

    And once in a while they load bulk data in those modules from 3rd party, or processed with offline tools (like Excel). With this base all those customers can benefit by simply having a CI - Component Interface for the same.

The process is simple as follows,

  1. You create and perfect your CI based on the data you want to load
  2. Build your own upload templates with defaults, formats, etc., 
  3. Complete the setup to create the File Layout (CSV or Fixed format)
    • Which can provide you the sample file in FIXED as well as XLSX format
  4. Perfect your data using Excel or Text pad
  5. Use the universal uploader process to load your content
 Few screens I have provided here to give a glimpse,

Template Design

Loads the CI Properties


Complete the Setup

Template/Sample Generation


Excel Template for CSV

Universal Uploader Process - Run Control

Process Logs

Benefits of this utility,
  1. No need of new process (or AE) to create every time you have different set or component to load
  2. Utility is capable of creating the File Layout automatically, so reduced technical work
  3. Sophisticated logs and information so that the technical or functional user aware what to do
  4. Multiple Templates for a single Component Interface to suite the business needs
You can find the project in my GitHub, (Better-Designs)

At present the process will load the data in FIXED file format only., I'm planning to improve the same to work for CSV and XML and also with CI logic based (so that you will not loose business validations). Any suggestion to improve the design and quality are welcomed.