Photo by author
# Introduction
As a data scientist or analyst, you know that understanding your data is the foundation of every successful project. Before you can build models, create dashboards, or create insights, you need to know what you’re working with. But exploratory data analysis, or EDA, is annoyingly repetitive and time-consuming.
For each new dataset, you write almost the same code to check data types, calculate statistics, plot distributions, and more. You need systematic, automated methods to quickly and fully understand your data. This article covers five Python scripts designed to automate the most important and time-consuming aspects of data exploration.
📜 You can find the scripts on GitHub..
# 1. Profiling Data
// Identifying the location of the pain
When you first open a dataset, you need to understand its basic properties. You write code to check data types, count unique values, identify missing data, calculate memory usage, and obtain summary statistics. You do this for each column, generating the same repeated code for each new dataset. This initial profiling can take an hour or more for complex datasets.
// Reviewing what the script does
Automatically generates a complete profile of your dataset, including data types, missing value patterns, cardinality analysis, memory usage, and summary statistics for all columns. Detects potential problems such as high cardinality categorical variables, constant columns, and data type mismatches. Generates a structured report that gives you a complete picture of your data in seconds.
// Explain how it works.
The script iterates through each column, determines its type, and calculates the relevant statistics:
- For numeric columns, it calculates the mean, mean, standard deviation, quartile, skewness, and kurtosis.
- For explicit columns, it indicates the unique values, mode, and frequency distribution.
This flags potential data quality issues such as columns with >50% missing values, categorical columns with too many unique values, and columns with zero variance. All results are compiled into an easy-to-read data frame.
⏩ Get the data profiler script.
# 2. Analyze and visualize distributions
// Identifying the location of the pain
Understanding how your data is distributed is essential to choosing the right transformations and models. You need to create histograms, box plots, and density curves for numerical characteristics and bar charts for specific characteristics. Creating these visualizations manually means writing plotting code for each variable, adjusting layout, and managing multiple figure windows. For datasets with dozens of features, this becomes cumbersome.
// Reviewing what the script does
Generates composite distribution concepts for all features in your dataset. Creates histograms with kernel density estimates for numerical features, box plots to show outliers, bar charts for specific features, and QQ plots to assess normality. Detects and highlights skewed distributions, multimodal patterns, and potential outliers. Organizes all plots in a neat grid layout with automatic scaling.
// Explain how it works.
The script separates numeric and explicit columns, then generates the appropriate visualizations for each type:
- For numerical features, this overlay creates subplots showing histograms with kernel density estimation (KDE) curves, annotated with skewness and kurtosis values.
- For clear characteristics, it produces sorted bar charts showing value frequencies.
The script automatically determines the optimal bin size, handles outliers, and uses statistical tests to flag distributions that deviate significantly from normality. All visualizations are created with consistent styles and can be exported as needed.
⏩ Get the distribution analyzer script.
# 3. Finding correlations and relationships
// Identifying the location of the pain
Understanding the relationships between variables is important but tedious. You will need to calculate correlation matrices, create scatter plots for promising pairs, identify multicollinearity problems, and detect non-linear relationships. Doing this manually requires creating dozens of plots, calculating various correlation coefficients e.g Pearson, Spearmanand Kendalland trying to find patterns in correlation heatmaps. The process is slow, and you often miss out on important relationships.
// Reviewing what the script does
Analyzes relationships between all variables in your dataset. Generates correlation matrices with multiple methods, creates scatter plots for highly correlated pairs, detects multicollinearity problems for regression modeling, and identifies non-linear relationships that may be missed by linear correlation. Creates visualizations that let you drill down into specific relationships, and flag potential problems like perfect correlations or redundant properties.
// Explain how it works.
The script computes correlation matrices using Pearson, Spearman, and Kendall correlations to capture different types of relationships. It generates an annotated heatmap that highlights strong correlations, then creates detailed scatter plots for feature pairs that exceed correlation thresholds.
To detect polynomials, it calculates Variance Inflation Factors (VIF) and identifies feature groups with high intercorrelations. The script also computes mutual information scores to capture non-linear relationships that lack correlation coefficients.
⏩ Get the Correlation Explorer script.
# 4. Detecting and analyzing outliers
// Identifying the location of the pain
Outliers can affect your analysis and models, but identifying them requires multiple methods. You need to check for outliers using various statistical methods, such as interquartile range (IQR), Z-score, and isolation forests, and visualize them with box plots and scatter plots. You then need to understand their impact on your data and decide whether they are true anomalies or data errors. Manually implementing and comparing multiple outlier detection methods is time-consuming and error-prone.
// Reviewing what the script does
Detects outliers using multiple statistical and machine learning methods, compares results from all methods to identify consensus outliers, generates visualizations showing outliers and patterns, and provides detailed reports on outlier characteristics. Helps you understand whether outliers are isolated data points or part of meaningful clusters, and evaluates their potential impact on downstream analysis.
// Explain how it works.
The script applies several outlier detection algorithms:
- The IQR method for univariate outliers
- Mahlanobs distance For multivariate outliers
- Z-scores and transformed z-scores for statistical outliers
- Forest of Solitude For complex anomaly patterns
Each method produces a set of flagged points, and the script creates a consensus score indicating how many methods flagged each observation. It develops side-by-side visualizations comparing detection methods, highlights flagged observations in multiple ways, and provides detailed statistics on outliers. The script also performs a sensitivity analysis that shows how outliers affect key statistics such as means and correlations.
⏩ Get the outlier detection script.
# 5. Analyzing missing data patterns
// Identifying the location of the pain
Missing data are rarely random, and it is important to understand the patterns of missingness in order to choose the right handling strategy. You need to identify which columns have missing data, detect patterns of missingness, visualize missingness patterns, and understand the relationship between missing values and other variables. Performing this analysis manually requires custom code for each dataset and sophisticated visualization techniques.
// Reviewing what the script does
Analyzes missing data patterns across your entire dataset. Identifies columns with missing values, calculates missing rates, and detects correlations in missing patterns. It then examines the types of missingness—Missing Completely at Random (MCAR), Missing at Random (MAR), or Missing Not at Random (MNAR)—and develops visualizations showing patterns of missingness. Provides recommendations for strategic response based on detected patterns.
// Explain how it works.
The script creates a binary missingness matrix that indicates where values are missing, then analyzes the matrix to detect patterns. It computes missingness correlations to identify features that are missing together, uses statistical tests to assess missingness patterns, and generates heatmaps and bar plots showing missingness patterns. For each column with missing data, it examines relationships between missingness and other variables using statistical tests and correlation analysis.
Based on the patterns found, the script recommends appropriate imitation strategies:
- Mean/median for MCAR numerical data
- Predictive imputation for MAR data
- A domain-specific approach to MNAR data
⏩ Get the missing data analyzer script.
# Concluding Remarks
These five scripts address the fundamental data exploration challenges that every data professional faces.
You can use each script independently for specific exploration tasks or combine them into a complete exploratory data analysis pipeline. The result is a systematic, reproducible approach to data exploration that saves you hours or days on every project and ensures you don’t miss essential insights into your data.
Have fun exploring!
Bala Priya c is a developer and technical writer from India. She loves working at the intersection of mathematics, programming, data science, and content creation. His areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding and coffee! Currently, she’s working on learning lessons and sharing her knowledge with the developer community, writing tutorials, how-to guides, opinion pieces, and more. Bala also creates engaging resource reviews and coding tutorials.