Lead Data Software Engineer

We are building cloud-scale data pipelines and platforms as a Lead Data Software Engineer to enable analytics, reporting, and AI-driven initiatives. You will collaborate with Product, Engineering, and Business teams to deliver reliable solutions using Azure Databricks, Spark, Kafka, and modern governance practices.

Design scalable batch and streaming data pipelines Build and fine-tune Databricks, Spark, and Kafka-based solutions Implement data quality, governance, and observability standards Develop data models and reporting foundations for analytics and AI initiatives Collaborate with Product, Engineering, and Business teams to deliver data-driven solutions Support enterprise reporting, dashboards, and self-service analytics platforms Leverage AI tools such as Claude Code, GitHub Copilot, and similar platforms for development, testing, and productivity improvements Apply MCP, prompt engineering, and multiple LLMs in AI-assisted development workflows Proven track record with 5+ years of experience in data engineering or a related field Hands-on proficiency in SQL, Python, PySpark, Scala, and Java Deep expertise in Apache Spark, Spark SQL, Kafka, and Delta Lake Strong experience with Azure Databricks, Unity Catalog, and Databricks Asset Bundles (DAB) Solid familiarity with Azure Data Factory (ADF), ADLS Gen2, Azure Key Vault, Azure Cosmos DB, and Azure Monitor Thorough knowledge of Data Modeling, Data Quality, Data Governance, and PII Handling Demonstrated background in Medallion Architecture (Bronze/Silver/Gold), CDC, and SCD Type 1 & Type 2 Practical competency in Azure DevOps, Git, CI/CD, and Gradle English proficiency at B2+ (Upper-Intermediate) level Familiarity with Tableau, ThoughtSpot, Cognos, and Data Visualization Best Practices Knowledge of Semantic Layer Concepts, AI/BI Dashboards, and Databricks Genie Understanding of scalable data platforms, analytics, and reporting solutions