Note: The job is a remote job and is open to candidates in USA. CareScout is on a mission to simplify and dignify the aging experience. They are seeking a highly skilled Lead Data Engineer to join their data and machine learning organization, responsible for building pipelines, models, and infrastructure that power analytics and operational data needs.
Responsibilities
- Design, build, and maintain scalable ETL/ELT pipelines using Spark, Python, SQL, and Databricks
- Implement reliable ingestion frameworks for batch and streaming data sources
- Ensure pipelines meet SLAs, data quality standards, and production-grade reliability
- Develop robust data models across raw, curated, and semantic layers using Delta Lake
- Create dimensional models, star schemas, and domain-layer datasets for analytics and ML
- Establish and maintain standards for schema design, metadata, and lineage
- Implement data validation, anomaly detection, SLAs, and documentation across pipelines
- Build automated tests, monitoring, and alerting for freshness, completeness, and accuracy
- Partner with platform teams to enhance observability and operational tooling
- Work closely with analysts to understand business KPIs and deliver high-quality curated datasets
- Partner with ML engineers and data scientists to build reusable feature pipelines
- Collaborate with data platform engineers to optimize compute, governance, and orchestration
- Optimize Spark jobs, SQL queries, cluster configurations, and storage patterns for performance and cost
- Improve reliability, reduce technical debt, and simplify complex pipelines
- Apply best practices for RBAC, data privacy, and PII handling using Unity Catalog
- Ensure adherence to compliance frameworks and documentation standards
- Stay current on modern data engineering patterns, Lakehouse architecture, orchestration, and best practices
- Explore new technologies that improve reliability, scalability, and developer productivity
Skills
- 7+ years of experience in data engineering or related roles
- Strong expertise with Python, SQL, Spark, and distributed data processing
- Hands-on experience with Databricks, Delta Lake, and Lakehouse architectures
- Deep understanding of ETL/ELT design, data modeling, and data quality practices
- Experience building scalable, production-grade data pipelines
- Experience collaborating with analytics, ML, and product teams
- Strong communication skills with the ability to clarify data requirements and explain technical decisions
Benefits
- Competitive Compensation & Total Rewards Incentives
- Comprehensive Healthcare Coverage
- Multiple 401(k) Savings Plan Options
- Auto Enrollment in Employer-Directed Retirement Account Feature (100% employer-funded!)
- Generous Paid Time Off – Including 12 Paid Holidays, Volunteer Time Off and Paid Family Leave
- Disability, Life, and Long Term Care Insurance
- Tuition Reimbursement, Student Loan Repayment and Training & Certification Support
- Wellness support including gym membership reimbursement and Employee Assistance Program resources (work/life support, financial & legal management)
- Caregiver and Mental Health Support Services
- This role is eligible to participate in a bonus incentive plan. Incentive compensation is based on individual and company performance and is not guaranteed.
Company Overview