Databricks AI/ML Engineer
Role Summary We are seeking an experienced Databricks AI/ML Engineer with approximately 8 years of experience to design, develop, and deploy scalable machine learning and data engineering solutions on Azure Databricks. The role combines end-to-end ML engineering, MLOps, and Lakehouse-based data engineering, with a focus on reliable, secure, and production-ready solutions that address business and operational needs. Key Responsibilities ā Design, develop, and deploy machine learning solutions using Azure Databricks and PySpark. ā Build end-to-end ML pipelines covering data preparation, feature engineering, model training, validation, deployment, and ongoing monitoring. ā Develop predictive and prescriptive analytics solutions for business and operational use cases. ā Implement MLOps practices for model versioning, monitoring, lifecycle management, and controlled releases. ā Create reusable notebooks, data-processing frameworks, ML accelerators, and engineering components. ā Process large-scale structured and unstructured datasets using Apache Spark and Databricks. ā Build and maintain ETL/ELT frameworks to ingest data from multiple enterprise source systems. ā Develop and optimize Delta Lake-based Lakehouse solutions and implement batch and real-time data-processing workflows. ā Automate model retraining and establish mechanisms to monitor model performance and operational health. ā Optimize model performance, scalability, processing efficiency, and overall solution reliability. ā Collaborate with business stakeholders, data engineers, data scientists, architects, and reporting teams to translate requirements into practical AI/ML and data solutions. ā Troubleshoot production issues, perform root-cause analysis, and implement corrective and preventive measures. ā Ensure data quality and adherence to enterprise data governance, security, privacy, and compliance standards. ā Participate in code reviews and promote sound engineering, testing, documentation, and development practices. ā Contribute to CI/CD automation and DevOps implementation for data and ML platforms. Required Skills and Experience ā Approximately 8 years of relevant experience in AI/ML engineering, data engineering, or related roles. ā Hands-on experience with Azure Databricks, Apache Spark, and PySpark. ā Experience building and operationalizing end-to-end machine learning pipelines. ā Strong understanding of feature engineering, model training, validation, deployment, and model performance monitoring. ā Practical knowledge of MLOps principles, model versioning, and ML lifecycle management. ā Experience with ETL/ELT design, data transformation, and large-scale data processing. ā Experience with Delta Lake and Lakehouse architecture concepts. ā Understanding of batch and real-time data-processing patterns. ā Familiarity with CI/CD, DevOps practices, code reviews, and automated deployment workflows. ā Ability to troubleshoot complex issues and optimize solutions for performance, scalability, and reliability. ā Strong collaboration and communication skills, with the ability to work across technical and business teams. Preferred Qualifications ā Experience delivering production-grade AI/ML solutions on Azure. ā Exposure to automated retraining, model drift detection, and model monitoring approaches. ā Experience working with enterprise data governance, access controls, and compliance requirements. ā Experience creating reusable frameworks or accelerators that improve development consistency and productivity.