Data Engineer (Data & Analytics Platform)
Key aspects of the project include:
Platform Design & Architecture:
Develop and implement data pipelines and architecture within Azure Data Lake, Azure Data Factory, and Azure Databricks.
Design scalable data models optimized for performance and reporting.
Data Ingestion & Transformation:
Build automated ETL/ELT pipelines using Databricks (PySpark) and Python.
Ingest data from various structured and unstructured sources (e.g., on-prem SQL Server, REST APIs, flat files).
SQL Development:
Write and optimize SQL queries and stored procedures for data wrangling and reporting.
Reporting & Visualization:
Collaborate with analysts to design and publish Power BI dashboards and reports.
Technical Stack:
Cloud Platform: Microsoft Azure
Data Storage: Azure Data Lake Gen2, Azure SQL Database
Data Processing: Azure Databricks, Azure Data Factory
Programming Languages: Python (PySpark), SQL
BI & Visualization: Power BI