Data Engineer
Data Engineer
Employee type
Contract
Salary
£450 - £500/day Remote Working
Category
IT
Job Description
We are looking for an experienced Data Engineer to support a data migration and integration programme within a relatively low maturity Azure environment.
The role will focus on designing and building robust data pipelines, migrating data from a wide range of legacy and enterprise source systems, and establishing scalable data flows into Azure Data Lake and Databricks. A key part of the role will also involve integrating data into a proprietary data transformation solution where applicable.
Key Skills & Experience
* Azure Data Engineering
* Azure Data Factory (ADF)
* Azure Data Lake
* Databricks
* Building ETL / ELT pipelines
* Large-scale data migration and integration
* Oracle and/or Oracle Fusion data sources
* SharePoint data extraction and integration
* Working with multiple and varied source systems
* Data transformation, mapping and validation
* Designing data pipelines in environments where architecture and processes are still developing
Key Responsibilities
* Design, build and maintain data migration and ingestion pipelines within Azure.
* Support the setup and development of the Azure Data Lake environment.
* Build integrations between source systems, Azure Data Lake and Databricks.
* Develop and manage pipelines using Azure Data Factory (ADF).
* Extract and migrate data from a variety of source systems, including:
* Oracle
* Oracle Fusion
* Informatica
* SharePoint, which will be an initial priority
* Additional enterprise and legacy systems as the programme develops
* Help define scalable data ingestion, transformation and migration patterns for future source systems.
* Support data quality, validation and troubleshooting throughout the migration process.
* Work closely with technical and business stakeholders to understand source data, target structures and transformation requirements.
* Help improve the overall maturity of the Azure data environment, introducing appropriate engineering standards and best practices