Data Engineer

Data Engineer – Azure & DatabricksExperience3–8 years of experience in Data Engineering, with strong hands-on experience in Azure and Databricks.Job SummaryWe are looking for an experienced Data Engineer with strong hands-on expertise in Microsoft Azure, Azure Databricks, and data engineering technologies. The candidate will be responsible for designing, developing, and maintaining scalable data pipelines and data solutions on Azure.Key ResponsibilitiesDesign, develop, and maintain data pipelines using Azure Data Engineering services.Develop and optimize data processing solutions using Azure Databricks and PySpark.Build and maintain ETL/ELT pipelines for batch and real-time data processing.Develop data ingestion workflows using Azure Data Factory (ADF).Work with Azure Data Lake Storage Gen2 (ADLS) for scalable data storage.Implement data transformation and cleansing using PySpark, Python, and SQL.Design and implement data models for analytical and reporting requirements.Optimize Databricks jobs, Spark workloads, SQL queries, and data pipelines for performance and cost.Implement data quality, validation, monitoring, and error-handling processes.Work with Delta Lake and implement Delta tables, schema evolution, partitioning, and optimization.Integrate data from multiple sources including databases, APIs, files, and cloud applications.Implement CI/CD processes for data pipelines using Azure DevOps or GitHub.Monitor production pipelines and troubleshoot data processing and performance issues.Collaborate with data architects, analysts, developers, and business stakeholders.Required Technical SkillsStrong experience in Data EngineeringHands-on experience with Microsoft AzureStrong experience with Azure DatabricksAdvanced PySparkStrong Python and SQLExperience with Azure Data Factory (ADF)Experience with Azure Data Lake Storage Gen2 (ADLS Gen2)Strong understanding of ETL/ELT conceptsExperience with Delta Lake