Automation & Data Engineering
Automated Mailing Data Pipeline — Python ETL
Cut execution time by 57% and boosted campaign efficiency by 27%
Bonus Photos / Details
The Problem
Mailing campaigns required that every hour from 08:00 to 20:00 (Monday to Saturday), someone would execute queries in PostgreSQL, wait for completion, export results to CSV, and import files to the dialer via SFTP. This process presented high risk of human error, campaign delays, and total dependence on technical operators. The project emerged during my area transition, requiring a solution that completely eliminated human execution.
The Solution
I developed an automated data pipeline that runs hourly: executes SQL queries in PostgreSQL, exports results to CSV, dynamically names files with date/time, automatically sends via SFTP to mapped folders, and the dialer automatically starts the campaign upon file detection. The process runs completely headless, without human intervention.
Tech Stack
Impact Generated
- Reduced execution time from 15-20 min → ~7 min per cycle (57% faster)
- Complete elimination of recurring manual work
- Fewer stopped campaigns
- Less idle time for robocall AIs
- Up to 27% increase in campaign efficiency
- Direct improvement in conversion rates and deals/base
Time Reduction
15-20 min → ~7 min per cycle
Efficiency
Up to 27% increase
Executions
Hourly, automated
Tech Stack
Overview
Automation,
Data Engineering
Technical Stack
Technical Challenges
- Learning and implementing Python integrations with PostgreSQL and SFTP
- Working around internal limitations (GPO)
- Ensuring stability and fault tolerance for connection failures
- Creating robust solution without dedicated infrastructure (running on local machine)