Position: Lead Data Engineer
Job Type: Full Time Contract
Location: Mississauga, ON (Hybrid)
Must Have: Azure DevOps and azure data engineering, SSIS, Databricks
Must Have
As our new Data Engineer, you will play a role in designing, developing, and maintaining our data architecture on the Azure platform. You can expect to collaborate with cross-functional teams, leverage
your technical proficiency, and contribute to the successful implementation of data solutions.
What will you be doing?
- Design and implement scalable, reliable, and high-performance data architectures on the Azure
- Cloud platform.
- Collaborate with data architects and other stakeholders to define data requirements and
- architecture standards.
- Develop end-to-end data pipelines, ensuring efficient and reliable data movement across
- various stages, from ingestion to consumption.
- Utilize Azure Data Factory, Azure Databricks, and other relevant services to orchestrate and
- optimize data workflows.
- Design and implement data models that support business requirements and enable efficient
- querying and analysis.
- Optimize data storage and retrieval processes for performance and cost-effectiveness.
- Integrate diverse data sources into a unified data platform, ensuring data consistency, accuracy,
- and reliability.
- Implement data integration patterns using Azure services such as Azure Data Factory, Azure
- Logic Apps, and Azure Functions.
- Ensure compliance with data governance and privacy regulations.
- Monitor data pipeline performance and proactively identify and resolve bottlenecks or issues.
- Implement performance tuning strategies to optimize data processing and reduce latency.
- Document data engineering processes, data flows, and system architecture for knowledge
- sharing and future reference.
What experience do you have?
- Bachelor’s degree in computer science, Information Technology, or a related field.
- Proven experience (3+ years) as a Data Engineer, with a focus on Azure Cloud technologies.
- In-depth knowledge of Azure services such as Azure Data Factory, Azure Databricks, Azure
- Synapse Analytics, and Azure Storage.
- 3-5+ years experience working with large volumes of data (structured and unstructured).
- Strong proficiency in programming languages such as Python, SQL, and familiarity with tools like
- Spark.
- Hands on experience on designing, developing, and implementing Enterprise Data Warehousing,
- ETL and real-time/batch data exchanges and working with related technologies such as SSIS,
- Azure Data Factory, Synapse Spark, Azure DevOps, Databricks.
- Strong SQL skills, knowledge of RDBMS and Data Lake concepts.
- Strong knowledge of data warehousing architecture, data mining methodologies and business
- intelligence.
- Knowledge of AI and ML developments/solutions/implementations is a plus.
- Knowledge of data security, compliance, and governance best practices.