Photo by author
# Introduction
As a data engineer, you will probably be (at least partially) responsible for your organization’s data infrastructure. You build pipelines, maintain databases, ensure data flows smoothly, and troubleshoot when things inevitably break. But here’s the thing: How much of your day is spent manually checking pipeline health, validating data loads, or monitoring system performance?
If you’re honest, it’s probably a huge chunk of your time. Data engineers spend many hours in their workday overseeing operational tasks, validating schemas, tracing data lineage, and responding to alerts—when they could be architecting better systems.
This article covers five Python scripts specifically designed to handle repetitive infrastructure and operational tasks that consume your valuable engineering time.
# 1. Pipeline Health Monitor
A word of pain: You have dozens of ETL jobs running on different schedules. Some run hourly, others daily or weekly. Checking that all of this has been completed successfully means logging into different systems, querying logs, checking timestamps, and piecing together what’s actually going on. By the time you realize you’re failing at a job, the downstream processes have already broken down.
What does the script do?: Monitors all your data pipelines in one place, tracks execution status, alerts on failures or delays, and maintains a historical log of job performance. Provides a consistent health dashboard that shows what’s running, what’s failed, and what’s taking longer than expected.
How it works: The script connects to your job’s orchestration system (eg Air flowor reads from log files), extracts execution metadata, compares against expected schedules and runtimes, and flags anomalies. It identifies patterns in success rates, average runtimes, and failures. Can send alerts via email or Slack when problems are detected.
⏩ Get the pipeline health monitor script
# 2. Change the schema validator and detector
A word of pain: Your upstream data sources change without warning. A column name changes, the data type changes, or a new required field appears. Your pipeline breaks, downstream notifications fail, and you’re probably struggling to figure out what changed and where. Schema drift is a very relevant problem in data pipelines.
What does the script do?: Automatically compares existing table schemas against baseline definitions, detecting any changes in column names, data types, constraints or structures. Generates detailed change reports and can enforce schema contracts to prevent breaking propagation through your system.
How it works: The script reads schema definitions from databases or data files, compares them against baseline schemas (saved as JSON), identifies additions, deletions, and modifications, and logs all changes with a timestamp. It can validate incoming data against expected schemas before processing and rejecting data that does not conform to it.
⏩ Get the schema validation script
# 3. Data lineage tracker
A word of pain: Someone asks “Where does this field come from?” or “What happens if we change this source table?” And you don’t have a good answer. You are trying to find the data flow in the SQL script, ETL code, and documentation (if it exists). Understanding dependency and impact analysis takes hours or days instead of minutes.
What does the script do?: Automatically maps data lineage by analyzing SQL queries, ETL scripts, and transformation logic. Shows you the complete path from the source system to the final tables, including all changes applied. Generates visual dependency graphs and impact analysis reports.
How it works: The script uses SQL parsing libraries to extract table and column references from queries, generate a directory of data dependencies, track the transformation logic applied at each step, and visualize the complete lineage. It can perform impact analysis that shows how changes in any source affect downstream objects.
⏩ Get the data lineage tracker script
# 4. Database Performance Analyzer
A word of pain: Queries are running slower than usual. Your desks are overflowing. Indexes may be missing or unused. You suspect performance problems but identifying the root cause means manually diagnosing, analyzing query plans, examining table statistics, and interpreting latent metrics. This is a time-consuming task.
What does the script do?: Automatically analyzes database performance by identifying slow queries, missing indexes, table bloat, unused indexes, and suboptimal configurations. Produces actionable recommendations with estimated performance impacts and provides the SQL needed to implement improvements.
How it works: Script query database system catalog and performance views (pg_stats for PostgreSQLfor , for , for , . SQL for Informas_schemaetc.
⏩ Get the Database Performance Analyzer script
# 5. Data Quality Claims Framework
A word of pain: You need to ensure data quality in your pipelines. What do you expect? Are there unexpected noles? Are there foreign key relationships? You write these checks manually for each table, scattered across scripts, with no consistent framework or reporting. When the checks fail, you get vague errors without context.
What does the script do?: provides a framework for Defining data quality claims as code: Row count thresholds, uniqueness constraints, referential integrity, value ranges, and custom business rules. Runs all assertions automatically, generates detailed fail notifications with context, and integrates with your pipeline orchestration to fail jobs when you don’t perform quality checks.
How it works: Scripts use a declarative syntax where you define criteria rules in plain Python or Yaml. It executes all assertions against your data, collects results with detailed failure information (which rows failed, what values were invalid), generates comprehensive reports, and can be integrated into pipeline DAGs to act as standard gates preventing bad data from spreading.
⏩ Get the Data Quality Claims Framework script
# wrap up
These five scripts focus on the core operational challenges that data engineers run into all the time. Here’s a quick recap of what these scripts do:
- Pipeline Health Monitor gives you centralized visibility into all your data jobs
- A schema validator breaks changes before they break your pipelines
- Data lineage tracker maps data flow and facilitates impact analysis
- A database performance analyst identifies bottlenecks and opportunities for optimization
- A data quality assertion framework ensures data integrity with automated checks
As you can see, each script addresses a specific pain point and can be used individually or integrated into your existing toolchain. So choose a script, test it in a non-production environment first, customize it for your specific setup, and gradually integrate it into your workflow.
Happy Data Engineering!
Bala Priya c is a developer and technical writer from India. She loves working at the intersection of mathematics, programming, data science, and content creation. His areas of interest and expertise include devops, data science, and natural language processing. She enjoys reading, writing, coding and coffee! Currently, she is working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces and more. Bala also engages resource reviews and coding lessons.