Data migration and data integration are one of those subject that you either love or hate – the Marmite (Vegimite) syndrome . I’m one of those people who love it.
Whilst working on a project that required both, I tried to explain the process to my wife – ‘cos she asked. As I went into the intricacies of staging and transformations, scheduled process etc.. I noticed I had lost her at the staging database – the clue was she was on facebook!
So, rewinding, I started again – using CRM, data migration terms and a real world example:
We are on a family holiday for 2 weeks, beach , sun, self-catering etc…We take suitcases full of clothes. The usual holiday approach of we take too much, clothes we won’t use, clothes we think we will need and the pair of shorts you live and die in.
When we get home , my daughter and I would like to find our clothes washed , ironed and placed in the correct room, wardrobe, chest of drawer and draw.
My daughter and me!.
Disparate Data sources.
On arriving home, neither myself or my daughter have any interest in the “laundry” cycle. We just assume there is a laundry fairy that makes things happen. So, as suitcases get emptied, piles of clothes appear – general in the same location of the opening of the suitcase.
My wife will collect all the piles of laundry and move these many piles to one big pile in the utility room.
Cleansing & De-Duplication.
From the big pile , she will sort the big pile into various smaller piles based on colour, material etc.. These piles enter the washing machine. The washing cycle takes a few hours and can only take the sorted pile of laundry of the same colour and type.
My daughter , in boredom mode from returning from a fun filled holiday, finds the PE bag in the corner of the room with 2 week old kit which she forgot about. This is mainly due to the excitement of the impending holiday. So, she dutifully bring the previously unknown bag of laundry down to the utility and dumps it in the middle of the other segmented piles of laundry.
The usual discussion happens along the lines of “Why didn’t you mention this earlier”, “I didn’t think it would matter”.
The effect of scope creep.
My wife, now has to sort the new bag of laundry and assess whether each item can be added to the existing pile or a new pile should be started. In some instances the new pile is not viable for a full load in the washing machine so she will make the decision to defer that pile for another week.
The net result of the sudden appearance of the PE kit is Dinner will be late “or go and get a take away”. The latter is more expense after a holiday.
The washing ironing cycle will take a few days to complete, largely due to the volume of clothes taken on holiday , bad planning on what was actually needed and even worse planning for the return trip when everything was thrown into the first opened suitcase.
However, during the weekend, we are still wearing clothes and still placing in the linen basket. Yes, we have been trained to place in the appropriate basket now instead of a pile on the floor. My wife periodically will collect the baskets content and move these to the now small piles of laundry in the utility room.
After a few days of the weekend, the washing , iron cycle is almost complete. Clothes are clean, ironed and have now been placed in the correct wardrobe etc. As part of the housework process, my wife will match socks – so I have a pair! and throw out old socks that are no longer in use.
The sign-off process of the bulk data load.
In this instance, wine. As the holiday bulk washing load has been completed.
Batch Data Integration as a scheduled task.
The daily repetitive weekly cycle of laundry basket to utility pile to washing machine to iron to wardrobe, happens as part of our weekly routine. It just gets done, its so mundane we don’t know we are doing it because it so automated.
On-Demand data integration.
The usual, 6am alarm clock as I get up to visit a client in London. “Do I have a shirt ?”. I won’t complete this discussion….
At the end of the “real” world example – it was clear , she had got it.