Lead Data Engineering

  • Intelletec
  • Nov 18, 2020
Permanent

Job Description

Locations: CT-Hartford, MA-Wellesley, NY-New York

Intelletec has partnered in helping the Nation's Premier Health Innovators in helping people on their path to better health. They are building a new health care model that is easier to use, less expensive, and puts the consumer at the center of their care!!

Manages and responsible for successful delivery of large scale data structures and Pipelines and efficient Extract/Load/Transform (ETL) workflows. Acts as the data engineering team lead for large and complex projects involving multiple resources and task.

Fundamental Components
  • Designs and develops complex and large scale data structures and pipelines to organize, collect and standardize data to generate insights and addresses reporting needs.
  • Writes complex ETL (Extract / Transform / Load) processes, designs database systems and develops tools for real-time and offline analytic processing.
  • Develop frameworks, standards & reference material for architecture and associated products.
  • Designs data marts and data models to support Data Science and other internal customers.
  • Behaves as mentor to junior team members to provide technical advice.
  • Applies knowledge of their systems and products to consult and advise on additional efforts across multiple domains spanning broader enterprise.
  • Collaborates with data science team to transform data and integrate algorithms and models into highly available, production systems.
  • Uses in-depth knowledge on Hadoop architecture, HDFS commands and experience designing & optimizing queries to build scalable, modular, and efficient data pipelines.
  • Uses advanced programming skills in Python, Java or any of the major languages to build robust data pipelines and dynamic systems.
  • Integrates data from a variety of sources, assuring that they adhere to data quality and accessibility standards. *Experiments with available tools and advises on new tools in order to determine optimal solution given the requirements dictated by the model/use case.

Background Experience
  • Strong collaboration and communication skills within and across teams.
  • Ability to communicate technical ideas and results to non-technical clients in written and verbal form.
  • Proven ability to create innovative solutions to highly complex technical problems.
  • Ability to leverage multiple tools and programming languages to analyze and manipulate large data sets from disparate data sources.
  • Ability to understand and build complex systems and solve challenging analytical problems.
  • Advanced knowledge in Java, Python, Hive, Cassandra, Pig, MySQL or NoSQL or similar.
  • Advanced knowledge in Hadoop architecture, HDFS commands and experience designing & optimizing queries against data in the HDFS environment.
  • Experience building and implementing data transformation and processing solutions.
  • Has in-depth knowledge of large scale search applications and building high volume data pipelines.
  • Experience with bash shell scripts, UNIX utilities & UNIX Commands.
  • 7 or more years of progressively complex related experience.
  • Master's degree or PhD preferred.
  • Bachelor's degree or equivalent work experience in Computer Science, Engineering, Machine Learning, or related discipline.