Senior Data Engineer

Job description
Seeking a Data Engineer with experience in Azure Data Factory, Azure DevOps, CI/CD, and Git version control



The Department

This Australian government department is crucial in developing policies and practices for the nation’s various sectors. It emphasizes sustainability and productivity, while also ensuring biosecurity to protect against pests and diseases.

The department supports scientific research and data collection to guide policy decisions and foster innovation. It also facilitates trade, helping Australian products reach international markets. One of its key goals is to achieve a $100 billion sector by 2030.



Job details

Looking for a Data Engineer to join the Digital Transformation Program to work across several data and analytics platforms. The candidate will develop and optimise data pipelines in Azure Databricks, with a strong focus on Python and SQL. The candidate will have expertise in Azure Data Factory, Azure DevOps, CI/CD, and Git version control, as well as a deep understanding of Kimball dimensional modelling and Medallion architecture. This role requires strong collaboration skills to translate business requirements into effective technical solutions.





Preferred Skills & Experience:

Strong proficiency in Python for data transformation, automation, and pipeline development.
Advanced SQL skills for query optimisation and performance tuning in Databricks Notebooks.
Hands-on experience with Azure Databricks for large-scale data processing.
Expertise in Azure Data Factory for orchestrating and automating data workflows.
Experience with Azure DevOps, including setting up CI/CD pipelines and managing code repositories with Git.
Strong understanding of Kimball dimensional modelling (fact and dimension tables, star/snowflake schemas) for enterprise data warehousing.
Knowledge of Medallion architecture for structuring data lakes with bronze, silver, and gold layers.
Familiarity with data modelling best practices for analytics and business intelligence.
Strong analytical and problem-solving skills with a proactive approach to identifying and resolving issues.
Excellent collaboration and communication skills, with the ability to engage both technical and business stakeholders effectively.




Essential criteria

Demonstrated experience in developing ETL/ELT processes for complex and/or large data movement, transformation and/or visualisations, particularly in a cloud environment.
Experience preparing data optimised for query performance in cloud computed engines. E.g. • Distributed computing engines (Spark) • Azure SQL • Python • R
Experience working collaboratively within agile development teams and with DevOps practices
Strong problem-solving skills, with the ability to analyse and resolve complex integration challenges.




Contract: 4-month Contract with 1 x 12 Months extensions.



Security Required: Must have a Baseline security clearance



Location -Canberra / Hybrid Based



How to Apply

Please upload your resume to apply. Candidates will need to be willing to undergo pre-employment screening checks which may include, ID and work rights, security clearance verification and any other client requested checks





Closing date: Sunday 16 February 2025 by +am







Call Joanne Finchett on 0480 002454 or email Joanne@whizdom.com.au for any further information