Data Cleansing

"How can I improve the integrity of my data?"

Data cleansing, data cleaning or data scrubbing is the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database. Used mainly in databases, the term refers to identifying incomplete, incorrect, inaccurate, irrelevant, etc. parts of the data and then replacing, modifying, or deleting this dirty data.

After cleansing, a data set will be consistent with other similar data sets in the system. The inconsistencies detected or removed may have been originally caused by user entry errors, by corruption in transmission or storage, or by different data dictionary definitions of similar entities in different stores.

Data cleansing differs from data validation in that validation almost invariably means data is rejected from the system at entry and is performed at entry time, rather than on batches of data.

The actual process of data cleansing may involve removing typographical errors or validating and correcting values against a known list of entities. The validation may be strict or fuzzy.

Some data cleansing solutions will clean data by cross checking with a validated data set. Also data enhancement, where data is made more complete by adding related information, is a common data cleansing practice.

Data cleansing may also involve activities like, harmonization of data, and standardization of data. Standardization of data is a means of changing a reference data set to a new standard, ex, use of standard codes.

IDN has a proven and effective methodology to assist clients in improving data quality on a standalone basis or as part of a data conversion project.

Data Cleansing

The majority of information companies use to make decisions is derived from data stored in application systems. Unless continual focus and investment is spent to maintain or improve data quality, it will degrade over time. The causes of data degradation are many:

  • Workarounds in systems are created to handle changing business requirements rather than investing to enhance a system
  • Limited or outdated training and procedures lead to data entry or maintenance errors
  • Employee turnover reduces the corporate "memory" about how data is used
  • Data quality is often sacrificed in the typically short timeframes of assimilating new systems resulting from mergers or acquisitions
  • Internal data conversion efforts to new systems are often under estimated and under delivered which can result in a future state that is worse than the current state

IDN assists companies with the cleansing of data by ensuring that data pass a set of quality criteria. Those include:

  • Validity
  • Decleansing
  • Accuracy
  • Completeness
  • Consistency
  • Uniformity
Contact us to further discuss any of your needs