Data analysis starts with dirty data – messy field data collected on the ground, often by hundreds of people. Someone has to sort through the mess and fix the small errors that have crept in.
Spotting Duplicate Entries
Spotting Duplicate Site Visits With Similar Details The data analysis company even spot duplicates for site visits that have been logged separately and include similar details. This could include site names, postcodes and dates for site visits.
Standardising Formats Across Records
Dates recorded in the format ‘12/03’ compared to ‘3rd December’ need to be converted to one date format. Similarly prices in pounds and pence that have been recorded inaccurately need to be rounded and recorded consistently. The same applies to free-text which has been completed in vastly different formats to other records.
Deciding What To Do With Gaps
The rule for missing fields has to be clearly defined and consistently applied. While some missing values might be negligible and therefore left as they are, others might be so critical that they require the data collector to go back to the site to try and obtain the missing information by follow-up.
Checking Outliers Against Expected Ranges
This kind of analysis can often expose obvious errors in data such as a price of £10,000 + £200 being logged for something that elsewhere in the data has been recorded at a price in the £20s or £30s, ten times higher than all the other figures for that type of thing.
Fixing Location Mismatches
When a GPS pin has been dropped by a device for a visit it might be found to be located several streets away from the address details completed for that visit by the user. Cross referencing the GPS pin with the postcode of the written address on the visit with a map will typically resolve such issues. Where it is not possible to establish a location for a visit it should be flagged for investigation rather than being guessed at.
Building Checks Into The Form Itself
However, the best way to prevent errors from occurring is to design the form to prevent errors from occurring in the first place.
If you want a Data Analysis Company, https://shepper.com/ is a good place to start.
Clean data is not a coincidence. It is created by applying a few good habits consistently.