Software Engineer II - Data Ingestion and Pipelines (London)

About Intelmatix

Intelmatix is a deep tech Artificial intelligence (AI) company founded in July 2021 by a group of MIT scientists with the vision of transforming enterprises to become cognitive. A cognitive enterprise is one that uses AI and Decision Intelligence in making their decisions. This leads to better business decisions with improved accuracy, reduced errors, and better outcomes across various aspects of the business.

Team Description

The Data Ingestion and Pipelines (DIP) team is responsible for designing, building, and optimizing scalable data systems that integrate and process data from a variety of sources. We develop efficient data pipelines that ensure seamless data flow into our data lake and transform it to support business intelligence and machine learning applications.

Our work spans the entire data lifecycle; from ingesting data from clients, to developing complex ETL processes, to building systems for scraping external datasets (e.g., traffic, weather, points of interest, etc.). The team is dedicated to delivering high-performance, reliable infrastructure that enables data-driven decision-making across the organization.

Job Summary

The Data Ingestion and Pipelines (DIP) team is seeking a highly skilled and experienced Software Engineer II. The ideal candidate will have a strong background in software development with a focus on building and optimizing data pipelines, ensuring data quality, and integrating data from various sources. As a Software Engineer II, you will play a key role in designing, developing, and maintaining scalable data infrastructure that supports our business intelligence and analytics efforts.

Key Responsibilities
• Data Pipeline Development: Design, develop, and maintain robust data pipelines and ETL processes to ingest, transform, and load data from diverse sources into our data warehouse.
• Data Quality and Governance: Implement and monitor data quality checks, ensuring accuracy, consistency, and reliability of data.
• Optimization: Optimize data processing workflows for performance, scalability, and cost-efficiency.
• System Monitoring and Maintenance: Monitor and maintain data systems, responding to SEVs or other urgent issues to ensure continuous operations.
• Collaboration: Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions that meet their needs.
• Documentation: Maintain comprehensive documentation for data pipelines, systems architecture, and processes.

Qualifications
• Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Experience: Minimum of 3 years of experience in software development.
• Technical Skills:
• Proficiency in programming languages such as Python, Java, or Scala.
• Knowledge of data modeling and schema design.
• Good system design skills of data-intensive applications
• Familiarity with SQL skills and relational databases (e.g., PostgreSQL, MySQL).
• Familiarity with at least one cloud platform (e.g., AWS, Azure, Google Cloud) and its data services.
• Analytical Skills: Strong problem-solving skills with a keen eye for detail and a passion for data.
• Communication: Excellent written and verbal communication skills, with the ability to articulate complex technical concepts to non-technical stakeholders.
• Team Player: Ability to work effectively in a collaborative team environment, as well as independently.

Preferred Qualifications
• Experience with big data technologies (e.g., Hadoop, Spark, Kafka).
• Experience with AWS and its data services (e.g. S3, Athena, AWS Glue).
• Familiarity with data warehousing solutions (e.g., Redshift, BigQuery, Snowflake).
• Knowledge of containerization and orchestration tools (e.g., Docker, ECS, Kubernetes).
• Knowledge of data orchestration tools (e.g. Prefect, Apache Airflow).
• Familiarity with CI/CD pipelines and DevOps practices.
• Familiarity with Infrastructure-as-code tools (e.g. Terraform, AWS CDK).
• Experience with machine learning pipelines and MLOps.

Employee Benefits

At Intelmatix, our benefits package is designed to meet the diverse needs of our employees, reflecting our dedication to their well-being and professional growth. Depending on your office location and specific needs, our benefits may include:
• Comprehensive Medical Insurance for you and your dependents
• In-Office Snacks Pantry
• Relocation Support
• Children's School Allowance
• Role-Related Training Support
• Wellness Programs
• Salary Advance for Housing Costs
• Travel Tickets
• Pension Contributions