Sponsored by
Hello friends!
Welcome to this week’s Sloth Bytes. I hope you had a great week!
Want to get the most out of ChatGPT?
ChatGPT is a superpower if you know how to use it correctly.
Discover how HubSpot's guide to AI can elevate both your productivity and creativity to get more things done.
Learn to automate tasks, enhance decision-making, and foster innovation with the power of AI.
Data Cleaning For Dummies
When I was studying machine learning in college, I unfortunately learned a hard truth:
You spend more time with data instead of building cool AIs.
I remember when the course first started and we were learning regression.
I had to use a dataset that the professor provided.
I was so excited to build my first ML model and the dataset we were given made sense, but then I took a closer look at the data:
“NaN”, ““, outliers, typos, and zeroes where there shouldn’t be.
That’s when I learned I had to clean my data.
I also learned the truth about data science and machine learning.
Instead of building cool AIs, you’re just the janitor of tech.
Cleaning up weird broken data….
Real Data Is Always Messy
Perfect datasets only exist in tutorials. Real data comes from:
Human input: Typos, skipped fields, jokes, skepticism
System errors: Sensor failures, database crashes
Integration issues: Different systems, different formats
Time: Requirements change, fields get repurposed
If your data looks perfect, you're not looking hard enough or someone already did the hard work.
Data Exploration First
Before cleaning, understand the dataset, the meaning of each field, and what “valid” actually means in that domain. Exploration should help you form cleaning rules—not give you permission to delete anything that looks weird.
# Basic exploration
print(df.info())
print(df.describe(include='all'))
print(df.nunique(dropna=False))
print(df.isna().sum())
# Inspect suspicious categorical values
print(df['color'].value_counts(dropna=False).head(20))Alright you’ve explored the data and you definitely see some problems.
What do you do?
Some Common Data Problems
Missing Values
# What you expect
age = [25, 30, 35, 40]
# What real source systems might contain
age = [25, None, 35, '', 'N/A', -999, 0]
# Not every odd value means the same thing.
# Some may mean missing; others may be valid or invalid by domain rule.Inconsistent Formats
# Date chaos
dates: ["2023-01-15", "01/15/2023", "Jan 15, 2023", "15-01-23"]
# Category madness
categories: ["Red", "red", "RED", "R", "crimson", "Red "]
Outliers and Impossible Values
These are REAL values from my survey. (Thanks…)
# Birthday values
birthdays: ["0001-01-17", "275760-08-06", "9999-09-09", "0006-09-24"]
# Ah yes, I love that I have readers born in the year 1 and the year 275760..
Here’s proof btw:
Data Cleaning Strategies
1. Handling Missing Values
Dropping missing rows:
# Drop only when that rule makes sense for the task.
df_clean = df.dropna(subset=['required_feature'])Dropping every row with any missing value can delete a huge and potentially biased slice of the dataset. First ask why the data is missing and whether missingness itself carries information.
Impute / represent missing values:
For some features, you can replace missing values with a statistic learned from training data, add an explicit “Unknown” category, or add a missingness indicator. The right choice depends on what the feature means and why values are absent.
# Simple exploratory/non-ML example
median_age = df['age'].median()
df['age'] = df['age'].fillna(median_age)
df['category'] = df['category'].fillna('Unknown')Median can be more robust than mean for skewed numeric data, but neither is automatically the correct imputation. Filling values changes the distribution and can hide meaningful missingness.
Model-ready imputation:
Mean/median imputation is not really “predicting” the missing value—it substitutes a summary statistic. More advanced imputers can estimate values from other features, but they add assumptions and can still introduce bias.
import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)
imputer = SimpleImputer(missing_values=np.nan, strategy='median')
X_train_imputed = imputer.fit_transform(X_train)
X_test_imputed = imputer.transform(X_test)Do not leak the test set into cleaning
Any preprocessing step that learns from data—imputation statistics, scaling parameters, vocabulary mappings, feature selection, PCA, target encoding, etc.—should be fit on training data and then applied to validation/test data.
Fitting those transformations on the full dataset leaks information from the test set into training and can make evaluation look better than reality. Pipelines help keep preprocessing and model fitting on the correct split.
2. Inconsistent Formats
For format issues, define a canonical representation and normalize toward it—but preserve the raw value somewhere when traceability matters.
Start with deterministic rules you can explain and reproduce:
# Normalize whitespace/case first.
df['color_clean'] = df['color'].str.strip().str.casefold()
# Explicit mappings are safer than guessing.
color_mapping = {
'r': 'red',
'crimson': 'red',
}
df['color_clean'] = df['color_clean'].replace(color_mapping)
# Parse dates and turn invalid values into NaT for review.
df['date_clean'] = pd.to_datetime(df['date'], errors='coerce')Fuzzy matching can help suggest likely corrections, but blindly auto-replacing values based on string similarity can merge genuinely different categories. Review thresholds/mappings, especially for names, locations, IDs, and sensitive data.
3. Dealing with Outliers
An outlier is not the same thing as an error. A huge transaction may be rare but real. A human birthday in the year 275760 is probably invalid. First decide whether the value is impossible, a measurement/data-entry error, or a legitimate extreme observation.
Use domain rules and provenance: acceptable ranges should come from the real meaning of the field, product rules, sensor specs, contracts, or source-system documentation—not a threshold chosen only because the chart looks nicer.
For legitimate outliers, options include keeping them, using robust statistics/models, transforming the feature, capping under a documented rule, or analyzing them separately. Deleting them solely because they hurt model performance can erase the exact cases your system most needs to handle.
If a value is genuinely impossible, prefer flagging/quarantining it or repairing it from an authoritative source when possible instead of silently dropping the row.
from datetime import date
# Example domain rule for a human birth date.
today = pd.Timestamp(date.today())
oldest_plausible = today - pd.DateOffset(years=125)
valid_birthdate = df['birthdate'].between(oldest_plausible, today)
invalid_rows = df.loc[~valid_birthdate]
# Review, repair, or quarantine invalid rows.Duplicates are more complicated than drop_duplicates()
Two identical rows might be accidental duplicates—or two legitimate events that happen to share the same values. Decide what uniquely identifies the real-world entity/event before deleting anything.
Duplicate customer records may require entity resolution across several fields, while duplicate payment/webhook events are better deduplicated using a stable provider event/idempotency ID than “these columns look the same.”
Keep the cleaning reproducible
Keep raw/source data immutable when possible.
Write transformations as code instead of manually editing spreadsheet cells.
Track data-quality checks: ranges, null rates, uniqueness, allowed categories, referential integrity, row counts.
Log how many rows/values each cleaning step changed or removed.
Version mappings/rules so future you can explain exactly how the cleaned dataset was produced.
When to Stop Cleaning
Perfect data doesn't exist. Know when "good enough" is actually good enough:
The remaining data quality is sufficient for the actual decision/model and known failure costs.
You have measured/documented residual issues instead of merely getting tired of cleaning.
Your validation/test evaluation reflects the same preprocessing future data will receive.
Cleaning rules are reproducible and monitored so new source data cannot silently drift into nonsense.
Data cleaning is not glamorous, but “clean” should mean valid, consistent, traceable, and appropriate for the task—not “we deleted all the weird rows until the chart looked pretty.”
Your model can only learn from the signals and mistakes present in the data pipeline. Fixing bad labels, leakage, biased missingness, broken joins, and invalid values usually matters more than choosing a slightly fancier algorithm.
Garbage in, garbage out.
If you want to keep learning
Machine learning explained — see what happens after your data stops looking like it was assembled by raccoons.
Federated learning explained — learn how training changes when raw user data stays on individual devices.
SQL explained — learn the database basics behind extracting, filtering, and fixing structured data.
Thanks for the amazing feedback! I’ll do better. As for the sloth facts, I ran out of them 😭
Thanks to everyone who submitted!
NeoScripter (PHP solution, respect), AspenTheRoyal, gcavelier, SauravChandra10, and RelyingEarth87.
Word Overlapping
Given two words, overlap them in such a way, morphing the last few letters of the first word with the first few letters of the second word.
Return the shortest overlapped word possible.
Examples
overlap("sweden", "denmark")
output = "swedenmark"
overlap("honey", "milk")
output = "honeymilk"
overlap("dodge", "dodge") "dodge"Notes
All words will be given in lowercase.
If no overlap is possible, return both words one after the other (example #3).
How To Submit Answers
Reply with
A link to your solution (github, twitter, personal blog, portfolio, replit, etc)
or if you’re on the web version leave a comment!
If you want to be mentioned here, I’d prefer if you sent a GitHub link or Replit!
That’s all from me!
Have a great week, be safe, make good choices, and have fun coding.
If I made a mistake or you have any questions, feel free to comment below or reply to the email!
See you all next week.
What'd you think of today's email?
Want to advertise in Sloth Bytes?
If your company is interested in reaching an audience of developers and programming enthusiasts, you may want to advertise with us here.
