Senior IT Consultant

Position Overview

We are seeking a highly skilled Senior IT Consultant to design and implement a secure, scalable Integrated Data Hub (IDH) using Databricks and Medallion Architecture. This role will be responsible for security design, access control configuration, data modeling, metadata management, and large-scale data processing across bronze, silver, and gold layers. Experience with FHIR data standards at the gold layer is considered a strong asset.

Key Responsibilities

Architect and build robust data pipelines across Medallion Architecture (Bronze/Silver/Gold layers) using Databricks.
Implement Delta Lake optimizations, Unity Catalog, and secure data access through RBAC and ACLs.
Profile large datasets to assess structure, completeness, and quality.
Design and optimize ETL/ELT pipelines (batch and streaming) using PySpark, SQL, and Delta Live Tables.
Develop and manage metadata-driven ingestion and transformation frameworks.
Integrate and manage REST APIs with downstream systems such as MDM and FHIR.
Configure and optimize Databricks jobs, clusters, and DevOps pipelines.
Lead logical and physical data modeling across all lakehouse layers.
Collaborate with business analysts and developers to translate requirements into technical solutions.
Guide sprint planning, backlog grooming, and act as a technical SME for data platform initiatives.
Required Skills & Experience

Technical Expertise

Data engineering experience in cloud environments.
Expert in Databricks, including:
Unity Catalog
Delta Lake
Auto Loader
PySpark
Deep understanding of Medallion Architecture and data lake design.
Proficient with Azure Data Lake, Azure DevOps, Git, and Postman.
Expert in SQL (complex queries, tuning, debugging).
Experience in CDC (Change Data Capture) strategies for batch and streaming.
Skilled in managing RBAC and fine-grained access controls.
Experience building metadata-driven pipelines using Python and PySpark.
Proficient in Databricks cluster configuration, version control, and environment management.
Capable of leading data modeling, metadata definition, and data governance strategies.
General Skills

Proven ability to translate business requirements into scalable data architectures.
Strong experience conducting data discovery, profiling, and quality assessments.
Experience leading data platform initiatives from discovery through deployment.
Effective communicator capable of working with technical and business stakeholders.
Strong analytical and diagnostic skills across infrastructure, ETL, and DB layers.
Passion for mentoring and establishing best practices and reusable frameworks.
Familiarity with Agile methodologies, sprints, user stories, and iterative delivery.
Skilled at assembling and refining Product Backlog Items, and collaborating with PMs and team leads.
Preferred Qualifications

Experience with FHIR data models and services.
Knowledge of enterprise data governance frameworks and best practices.
Job Types: Full-time, Permanent, Fixed term contract
Contract length: 12 months

Pay: From $70.00 per hour

Expected hours: 36.25 per week

Schedule:

Monday to Friday
Application question(s):

How many years of hands-on experience do you have with Databricks, including Unity Catalog, Delta Lake, Auto Loader, and PySpark?
How many years have you worked with Medallion Architecture (Bronze/Silver/Gold) in Databricks?
How many years of experience do you have conducting data profiling to assess structure, completeness, and data quality?
How many years of experience do you have designing and managing ETL pipelines, including Change Data Capture (CDC)?
Work Location: Hybrid remote in Toronto, ON

Application deadline: 2025-07-16
Expected start date: 2025-07-21