Company:
Diverse Lynx
Location: Dallas
Closing Date: 19/10/2024
Hours: Full Time
Type: Permanent
Job Requirements / Description
The Data Engineer Tech Lead will play a crucial role in designing, developing, and maintaining the ETL processes and systems that facilitate data integration and transformation within the organization. The person will analyze business requirements, design efficient data workflows, and implement scalable and robust ETL solutions. This position requires strong technical skills, a deep understanding of data warehousing concepts, and expertise in ETL and reporting tools and technologies.
Requirements:
A bachelor s degree in computer science is required.
Proven experience as an ETL Engineer or in a similar role, with a minimum of 10+ years of experience designing and developing ETL processes.
7+ years of experience in lead or senior roles; demonstrating ability to lead data engineering projects end-to-end.
Strong expertise in ETL tools and technologies, such as ADF, Databricks, SQL, Python, SSIS, and SSRS.
Proficiency in SQL and experience working with relational and cloud databases, preferably Snowflake and SQL Server.
Demonstrated experience building, optimizing, and supporting large-scale data platforms.
Software engineering mindset, leading with the principles of source control, infrastructure as code, testing, modularity, automation, CI/CD, and observability.
Strong knowledge and understanding of the modern data platform and its key components - ingestion, transformation, curation, quality, governance, and delivery.
Experience working with a wide variety of source systems, building data pipelines to ingest both streaming and batch data, and delivering clean data for downstream usage in reporting, analytics, data science, and applications.
Strong knowledge of data modeling techniques (3NF, Dimensional, Vault), data lakes, data warehouses, data marts, design patterns (lambda, kappa, medallion, etc.), and agile development.
Familiarity with cloud-based data platforms (e.g., AWS, Azure) and related services (e.g., S3, Redshift, Big Query) is desirable.
In-depth knowledge of data integration patterns, data modeling, and database concepts.
Strong analytical and problem-solving skills, with the ability to troubleshoot and resolve complex data issues.
Familiarity with business intelligence visualization tools such as Looker, Tableau, Power BI, etc.
Prior experience working in Enterprise Agile/Scrum teams.
Excellent communication and interpersonal skills, with the ability to collaborate effectively with technical and non-technical stakeholders.
Strong attention to detail and commitment to delivering high-quality solutions.
Ability to work independently and manage multiple priorities in a fast-paced environment.
Key Responsibilities:
ETL Solution Design: Collaborate with stakeholders, including business analysts, data architects, and developers, to understand data integration requirements and design optimal ETL solutions. Ensure adherence to best practices and industry standards in data transformation and loading processes.
Development and Implementation: Develop, test, and deploy ETL processes using various ETL tools and technologies. Create scalable, efficient, and reliable data workflows that handle complex data transformation requirements. Optimize performance and ensure data quality throughout the ETL lifecycle.
Data Modeling: Collaborate with data architects to define data models and design schemas that support efficient data storage and retrieval. Implement appropriate data validation and cleansing routines to ensure data consistency, integrity, and accuracy.
Data Integration: Integrate data from multiple sources, such as databases, files, APIs, and external systems, into a centralized data warehouse or data lake. Implement data extraction strategies that maximize efficiency and minimize impact on source systems.
Troubleshooting and Performance Tuning: Identify and resolve data extraction, transformation, and loading issues. Perform root cause analysis and implement performance optimization techniques to enhance overall ETL process efficiency.
Documentation: Create and maintain technical documentation, including ETL specifications, data mappings, and process workflows
Collaboration and Leadership: Collaborate with cross-functional teams, including report developers, database administrators, and business analysts, to ensure successful project delivery. Provide technical guidance and mentoring to junior ETL developers, promoting knowledge sharing and continuous improvement.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
The Data Engineer Tech Lead will play a crucial role in designing, developing, and maintaining the ETL processes and systems that facilitate data integration and transformation within the organization. The person will analyze business requirements, design efficient data workflows, and implement scalable and robust ETL solutions. This position requires strong technical skills, a deep understanding of data warehousing concepts, and expertise in ETL and reporting tools and technologies.
Requirements:
A bachelor s degree in computer science is required.
Proven experience as an ETL Engineer or in a similar role, with a minimum of 10+ years of experience designing and developing ETL processes.
7+ years of experience in lead or senior roles; demonstrating ability to lead data engineering projects end-to-end.
Strong expertise in ETL tools and technologies, such as ADF, Databricks, SQL, Python, SSIS, and SSRS.
Proficiency in SQL and experience working with relational and cloud databases, preferably Snowflake and SQL Server.
Demonstrated experience building, optimizing, and supporting large-scale data platforms.
Software engineering mindset, leading with the principles of source control, infrastructure as code, testing, modularity, automation, CI/CD, and observability.
Strong knowledge and understanding of the modern data platform and its key components - ingestion, transformation, curation, quality, governance, and delivery.
Experience working with a wide variety of source systems, building data pipelines to ingest both streaming and batch data, and delivering clean data for downstream usage in reporting, analytics, data science, and applications.
Strong knowledge of data modeling techniques (3NF, Dimensional, Vault), data lakes, data warehouses, data marts, design patterns (lambda, kappa, medallion, etc.), and agile development.
Familiarity with cloud-based data platforms (e.g., AWS, Azure) and related services (e.g., S3, Redshift, Big Query) is desirable.
In-depth knowledge of data integration patterns, data modeling, and database concepts.
Strong analytical and problem-solving skills, with the ability to troubleshoot and resolve complex data issues.
Familiarity with business intelligence visualization tools such as Looker, Tableau, Power BI, etc.
Prior experience working in Enterprise Agile/Scrum teams.
Excellent communication and interpersonal skills, with the ability to collaborate effectively with technical and non-technical stakeholders.
Strong attention to detail and commitment to delivering high-quality solutions.
Ability to work independently and manage multiple priorities in a fast-paced environment.
Key Responsibilities:
ETL Solution Design: Collaborate with stakeholders, including business analysts, data architects, and developers, to understand data integration requirements and design optimal ETL solutions. Ensure adherence to best practices and industry standards in data transformation and loading processes.
Development and Implementation: Develop, test, and deploy ETL processes using various ETL tools and technologies. Create scalable, efficient, and reliable data workflows that handle complex data transformation requirements. Optimize performance and ensure data quality throughout the ETL lifecycle.
Data Modeling: Collaborate with data architects to define data models and design schemas that support efficient data storage and retrieval. Implement appropriate data validation and cleansing routines to ensure data consistency, integrity, and accuracy.
Data Integration: Integrate data from multiple sources, such as databases, files, APIs, and external systems, into a centralized data warehouse or data lake. Implement data extraction strategies that maximize efficiency and minimize impact on source systems.
Troubleshooting and Performance Tuning: Identify and resolve data extraction, transformation, and loading issues. Perform root cause analysis and implement performance optimization techniques to enhance overall ETL process efficiency.
Documentation: Create and maintain technical documentation, including ETL specifications, data mappings, and process workflows
Collaboration and Leadership: Collaborate with cross-functional teams, including report developers, database administrators, and business analysts, to ensure successful project delivery. Provide technical guidance and mentoring to junior ETL developers, promoting knowledge sharing and continuous improvement.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Share this job