Data cleaning is the unsung hero of the data science process. It’s often said that 80% of a data scientist’s time is spent cleaning data, and only 20% analyzing it. While it may not be glamorous, poor data cleaning can derail entire projects, lead to misleading insights, or even cost businesses a substantial amount of money.
In this post, we’ll explore some common data cleaning mistakes and, more importantly, how you can avoid them.
- Deleting Missing Data Without Thinking: A normal reaction to encountering missing values is to drop rows or columns that have null values. While this approach is easy, it can lead to data loss and inaccurate data. Here are some ways to avoid this mistake:
- Analyze the missing data. Check if the missing data occurs randomly or if it follows a specific pattern
- Try filling in missing values using the mean, median, mode, or even model-based imputation
- Compare models with and without imputed values to see the effect.
- Ignoring Outliers: Outliers can make analysis and models biased but ignoring them altogether or removing them is also risky. Hence, they have to be carefully evaluated to understand the next line of action. This mistake can be avoided by following the measures stated below:
- Overlooking Duplicates: Failing to check for and remove duplicate records can inflate data and skew analysis. How to avoid this mistake:
- Use .duplicated() in Pandas to find duplicates.
- Confirm whether the duplicates are genuine errors or intentional entries before removing them.
- Incorrect Data Types: Treating numeric values as strings, dates as objects, or categories as text can hinder analysis and cause bugs. How to avoid the mistake:
- Inspect your dataframe using .info() or .dtypes in pandas.
- Convert data types to the correct data type using functions from pandas like pd.to_numeric() (to convert to numeric values), pd.to_datetime() and astype().
- Ignoring Inconsistent Categorical Values: Having categories like “Male”, “male”, “M”, or “New York”, “new york”, “NYC” can fragment your data.
- Standardize casing and spelling by converting data to a uniform case.
- Map values manually or with a dictionary
- Use fuzzy matching libraries for large datasets
- Cleaning Before Understanding the Data: Jumping into cleaning without understanding the context or business problem can lead to removing important information.
- Start with EDA (Exploratory Data Analysis): Use plots, summaries, and statistics to get familiar.
- Collaborate with domain experts to avoid deleting meaningful anomalies
Data cleaning isn’t just a technical step, it’s a critical thinking exercise. The quality of your data directly influences the quality of your insights. By avoiding these common mistakes and adopting best practices, you’ll build more reliable models, uncover more accurate insights, and save yourself a lot of frustration.
