Data Engineer

The Senior Data Engineer plays a critica role in designing, developing, and maintaining scalable data infrastructure and pipelines. This position supports data-driven decision-making across the organization by ensuring data quality, accessibility, and performance.

Roles and Responsibilities

• Data Pipeline Development: Design, build, and maintain robust, scalable data pipelines using tools like Apache Airflow, Kafka, and Spark.
• Data Architecture: Develop and optimizedata models, data warehouses, and ETL processes to support analytics and machine learning initiatives.
• Collaboration: Work closelywith data scientists, analysts, and software engineers to understand data requirements and deliver tailored solutions.
• Data ǪualityG Governance: Ensure data integrity, implement validation checks, and enforce data governance policies.
• Performance Optimization: Monitor and improvedata processing speed,system uptime, and query performance.
• Leadership G Mentorship: Guide junior engineers, promote best practices, and foster a culture of continuous learning.
• Innovation G Research: Stay updated with emerging technologies in AI, automation, and data streaming; lead initiatives to integrate them into existing systems.

Profile

• Bachelor's or Master's in computer science, data science, information technology, or a related field, with 6+ years of experience.
• Proficiency in SǪL, Python, and ETL/ELT tools
• Experience in Microsoft solutions are as follows:
• Azure Data Factory
• Microsoft Fabric
• Azure Synapse Analytics
• Azure Databricks
• Power BI
• Azure SǪL Database/ SǪL Server
• Azure Purview
• Experience with big data technologies (Hadoop, Spark)
• Familiarity with cloud platforms(Azure, AWS, and GCP)
• Strong understanding of data modelling, data warehousing, and stream processing
• Knowledge of data security, compliance, and governance
• Knowledge of CI/CD pipelines using Azure DevOps
• Understanding of data security and compliance(e.g., GDPR, HIPAA)