Job Description
The Data Engineer will design, develop, and maintain scalable data pipelines and workflows using Python, PySpark, and cloud technologies.
This role involves managing data storage solutions, orchestrating ETL processes, and deploying containerized applications to support enterprise data initiatives.
Key Responsibilities
- Develop, construct, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, PySpark, and SQL.
- Manage and optimize data storage solutions in Snowflake and Data Lakes.
- Integrate data from multiple sources while ensuring data quality, integrity, and security.
- Design and implement automated workflows using Azure Data Factory or similar orchestration tools.
- Deploy and manage containerized applications and services on OpenShift.
- Collaborate with Data Scientists, Analysts, and stakeholders to understand data requirements.
- Monitor and troubleshoot data pipelin...