Staff Software Engineer

Staff Software Engineer, Data Ingestion

The Staff Software Engineer, Data Ingestion will be a critical individual contributor responsible for designing, developing, and maintaining robust and scalable data pipelines. This role is at the heart of our data ecosystem, delivering new analytical software solutions to access timely, accurate, and complete data for insights, products, and operational efficiency.

Key Responsibilities:
• Data Ingestion Pipelines: Design, develop, and maintain high-performance, fault-tolerant data ingestion pipelines using Python.
• Integrations: Integrate with diverse data sources (databases, APIs, streaming platforms, cloud storage, etc.).
• Data Transformation: Implement data transformation and cleansing logic during ingestion to ensure data quality.
• Pipeline Monitoring: Monitor and troubleshoot data ingestion pipelines, identifying and resolving issues promptly.
• Database Collaboration: Collaborate with database engineers to optimize data models for fast consumption.
• Technology Evaluation: Evaluate and propose new technologies or frameworks to improve ingestion efficiency and reliability.
• Self-Healing Mechanisms: Develop and implement self-healing mechanisms for data pipelines to ensure continuity.
• SLAs and SLOs: Define and uphold SLAs and SLOs for data freshness, completeness, and availability.
• On-Call Rotation: Participate in on-call rotation as needed for critical data pipeline issues.

Key Skills:
• Engineering Background: 8+ years of experience, ideally with an engineering background, working in software product companies.
• Python Expertise: Extensive Python expertise: extensive experience in developing robust, production-grade applications with Python.
• Data Collection & Integration: Proven experience collecting data from various sources (REST APIs, OAuth, GraphQL, Kafka, S3, SFTP, etc.).
• Distributed Systems & Scalability: Strong understanding of distributed systems concepts, designing for scale, performance optimization, and fault tolerance.
• Cloud Platforms: Experience with major cloud providers (AWS or GCP) and their data-related services (e.g., S3, EC2, Lambda, SQS, Kafka, Cloud Storage, GKE).
• Database Fundamentals: Solid understanding of relational databases (SQL, schema design, indexing, query optimization). OLAP database experience is a plus (Hadoop)
• Monitoring & Alerting: Experience with monitoring tools (e.g., Prometheus, Grafana) and setting up effective alerts.
• Version Control: Proficiency with Git.
• Containerization: Experience with Docker and Kubernetes.
• Streaming Technologies: Experience with real-time data processing using Kafka, Flink, Spark Streaming.