Senior Data Engineer - Azure & Databricks Expert

Job description
Senior Data Engineer – Azure & Databricks Expert (Baseline Clearance Required)

Location: Canberra, ACT
Security Clearance: Must have an active Baseline Clearance or be eligible to obtain one
Contract Duration: 4 months (with a potential 12-month extension)
Rate: Competitive – based on experience
Working Arrangements: Hybrid (minimum 3 days per week in the office)
Maximum Hours: 40 hours per week

About the Role
We are seeking a Senior Data Engineer to join a federal government digital transformation program. The successful candidate will be responsible for designing, developing, and optimising data pipelines using Azure Databricks, with a focus on Python, SQL, and Azure cloud technologies.

This role requires expertise in Azure Data Factory, Azure DevOps, and CI/CD workflows, with a strong foundation in Kimball dimensional modelling and Medallion architecture. You will collaborate with business stakeholders and cross-functional teams to translate business requirements into scalable, efficient, and high-performing data solutions.

Key Responsibilities
- Develop, optimise, and maintain data pipelines using Python and SQL within Azure Databricks Notebooks
- Design and implement ETL/ELT workflows in Azure Data Factory, ensuring efficient data transformation and loading
- Apply Kimball dimensional modelling and Medallion architecture for structured, scalable data solutions
- Work closely with stakeholders to translate business requirements into effective technical solutions
- Implement and maintain CI/CD pipelines using Azure DevOps, ensuring automated deployments and Git version control
- Monitor, troubleshoot, and optimise Databricks jobs and queries for performance and efficiency
- Collaborate with data analysts and BI teams to deliver high-quality datasets for analytics and reporting
- Ensure data governance, security, and privacy best practices are followed
- Contribute to code quality improvements through peer reviews and best practices

Preferred Skills & Experience
- Strong proficiency in Python for data transformation, automation, and pipeline development
- Advanced SQL skills for query optimisation and performance tuning
- Hands-on experience with Azure Databricks for large-scale data processing
- Expertise in Azure Data Factory for orchestrating and automating data workflows
- Experience with Azure DevOps, including setting up CI/CD pipelines and managing Git repositories
- Strong understanding of Kimball dimensional modelling (fact and dimension tables, star/snowflake schemas) for enterprise data warehousing
- Knowledge of Medallion architecture for structuring data lakes with bronze, silver, and gold layers
- Familiarity with data modelling best practices for analytics and business intelligence
- Strong analytical and problem-solving skills, with a proactive approach to resolving data integration challenges
- Excellent communication and teamwork skills, ensuring alignment between technical and business requirements

How to Apply

Please apply online or send your resume to George@LocalWorkforceHire.com.au or call him for a chat on 0433 704 567