Sr Data Engineer 2-3 days on site, Toronto, ON Duration: 156 weeks
1.0 Description of Assignment
Data Engineer will be responsible for modeling, designing and developing data pipelines, and analytics ready data sets and business intelligence solution(s).
He/she will follow design & development methodologies and documentation standards.
Hybrid work environment according to office policy. Currently at Union Station (61 Front St W, Toronto) with minimum 2 days on site.
Qualifications
Mandatory Requirements 5+ years of experience in:
Database development or data engineering for enterprise level solutions
Hands-on experience working with:
Microsoft Azure Synapse and/or SQL Data Warehouse and SQL Server 2016 or later.
Microsoft Azure Data Factory and other related Azure Services
Azure Databricks
CI/CD pipelines (Azure DevOps)
SQL and Python programming
Developing Microsoft Power BI reports or other data visualization tools
Other Certifications / Skills
A university degree in Computer Science, or the equivalent, combined with directly related experience in a business or computer related discipline.
Optimize, analyze and improve data integration packages
Expert understanding and enterprise scale experience working with Data Warehouse (DW) and Business Intelligence (BI) implementations
Strong understanding of data modelling techniques and data warehouse/data lake design
Experience with data quality, data cleansing and data stewardship processes and techniques
Ability to solve complex data problems by analysis, develop appropriate solutions and clearly communicate those solutions to the team.
Ability to work efficiently and provide accurate time estimates for work to be undertaken.
Well-developed interpersonal, oral and written communication skills, strong analytical and problem-solving skills.
3.0 Assignment Duties
Work with the Enterprise Data project team in designing, developing and maintaining the client’s enterprise data warehouse, data marts, data lakes
Responsible for designing, developing and documenting SSIS and/or Azure Data Factory (ADF) packages to integrate data from multiple systems
Responsible for designing, developing and documenting complex T-SQL, stored procedures, views, triggers, indexes, etc.
Develop and maintain Databricks notebooks and jobs using Python (PySpark) and SQL for batch and streaming workloads
Build curated, analytics-ready datasets following layered patterns (e.g., bronze/silver/gold) and data product principles.
Implement Delta Lake best practices
Responsible for designing and implementing data quality processes
Assist in the data modelling of the data warehouse/data mart/data lake solution(s)
Advanced data analysis to provide meaningful data driven insights
Work closely with the business analyst(s), and other team members, to understand requirements
Convert business requirements to technical specifications.
Perform knowledge transfer meetings and sessions.
Research, propose, document, and share technical solutions with the team.
Recommend and implement improvements to data quality, automation, and workflows
Provide accurate time estimates for work to be undertaken
4.0 Deliverables
Detailed technical design and functional specifications.
Completed system components 'unit tested' and ready for System, QA and User Acceptance testing.
Technical support during the various stages of testing (system integration, QA, User Acceptance)