The world is undergoing a data revolution, with data now viewed as the most valuable commodity for businesses. Because of this, there’s no end of specialties related to data, and there’s even a term for the complex system in which we interact with it – the Datasphere. It is estimated that by 2025 75% of the world will be connected and interacting with data every day.
Furthermore, IDC reports that the enterprise role as a data steward is continuing to grow, as from 2019 more data is being stored in the enterprise core than in all the world’s end points.
This means businesses need to keep up, and a data migration may be required. This could be in the form of an entire system overhaul, the merger of two data systems, or an upgrade of your current systems. Whatever the reason, moving data from one environment to another takes expert knowledge. Not sure where to start? Let us take you through the key terms involved in a data migration.
Data Conversion:
The transformation of data from one format to another.
Data Mapping:
This is the process of identifying the fields that you want to migrate from the source system and mapping them to the corresponding fields in the target system. There might be fields in the source system that you require that do not have a corresponding field in the target system. This will need a custom field to be created in the target system. Sometimes the target system will have a required (mandatory) field that is not a field within the source system.
Data Migration:
The process of transferring data from a source system to a target system or from one technology to another. Our approach to data migration involves three clear steps – Discover, Migrate, Reconcile.
Data Verification:
The process of sanity checking the data. This is often done pre and post migration. Working with a Subject Matter Expert (SME) from the business prior to the migration can help with defining the logic used in the any required data transformation.
ETL process:
A three-stage process within a data migration that includes Extract/Transform/Load (ETL).
a) Extract – Identifying the source data required for the migration and extracting the data. The extraction of the data is typically done with SQL scripts.
b) Transform – Often the target system requires the source data to be transformed into a new format or data cleansing is required to remove data anomalies.
c) Load – Once the transformation is complete the data can be loaded into the target system. There are various techniques that can be used including SQL scripts, load templates and target system load tools.
Data Migration Strategies:
Different approaches to how data migration is done. These could be:
a) Big bang migration – as it sounds, the operation is done as a single event, typically over a weekend.
b) Trickle migration – also known as iterative migration, where the entire process is broken down into sub migrations with their own scope, goals, timelines and quality checks.
c) Hybrid – To reduce the risk of a big bang migration, especially if the cut-over window from source to target is small or it is a very large and complex migration, a combination of a bulk load with delta loads can be used.
Data Migration Testing:
This is required throughout the data migration process and should include:
a) Pre-migration testing – this aims to test the requirements against the source data and test the data mappings and applied business rules are correct.
b) Migration testing – this involves the testing of the migration ETL process.
c) Post-migration testing – structured testing routines need to be developed to test that the migrated data in the target system is correct and complete.
Types of Data Migration:
These depend on the business problem being solved and include but are not limited to the following:
a) Application Migration moves an application program from one environment to another. This may include moving the entire application from an on-premises IT centre to a cloud.
b) Cloud Migration moves an organisation’s applications — including data, services, processes, and other business components — to a cloud computing environment.
c) System Migration or system changeover is the process of moving data and applications from one system to another. For example, from Oracle to SAP or Microsoft Dynamics to Salesforce.
Reconciliation:
Compares target data with source data to make sure the data has been transferred correctly and that data integrity has been maintained
Data Cleansing:
This is the process of fixing data errors and improving the quality of the data. There are 3 places that the data can be cleaned:
a) Clean/fix data in the source system.
b) During the transformation process.
c) Clean/fix the data in the target system.
A data migration involves a lot of moving parts. The approach needed depends on the type of business you are in, the systems and applications involved, and the required data use. It’s not something a business does multiple times, and it’s also not something that can be done without experience and meticulous focus. Harness the power of your data by engaging a specialist who’s done it before. We’re experienced across multiple systems and can deliver your data migration efficiently and with fully reconciled data – guaranteed.