Job Description
DESCRIPTION
We are currently looking for a Data Engineer for a client in Toronto. They are looking for someone with experience working with Azure, DataBricks and experience working in the Banking/Finance sectors.
RESPONSIBILITIES
- Analyze and understand business requirements related to data and information, with a particular focus on the financial services industry.
- Design data integration and ETL processes using tools such as Databricks, Spark, Hadoop, and Hive, following a holistic, scalable, pragmatic, and effective approach to data processing.
- Ingest and transform data in Databricks for batch and near real-time processing using technologies such as PySpark and SparkSQL, while leveraging orchestration tools like Azure Data Factory within the broader Microsoft Azure ecosystem. Design and implement data pipelines integrated with Continuous Integration/Continuous Delivery (CI/CD) frameworks as required (DevOps). ...