Data Engineer (Analyst, IT)

If you are passionate about addressing challenging issues in higher education through the use of data engineering and educational data mining techniques, we would like to talk to you about a role within the Data and Analytics Practice in Duke University’s Office of Information Technology.

Duke has been named as a best place to work by several publications and organizations, including Forbes Magazine, National Center for Transit Research, Chronicle of Higher Education, Computerworld, The Scientist, and the American Association of Retired Persons, among others. The organizations recognizing Duke represent a diverse range of interests, from family-friendly and preventive health to the needs of IT professionals and academic researchers.

The Data Analytics Practice with Duke’s Office of Information Technology provides Data Engineering, Data Visualization, and Data Science support for the university’s institutional, academic, and research needs. The Data Engineering position serves as a key member of the Data Analytics team, working across a breadth of projects. We offer a hybrid work environment that combines the flexibility of remote work with the benefits of occasional in-person collaboration. Candidates for this position must reside in North Carolina. Details on Duke’s remote work policy can be found at https://remotework.duke.edu/guidelines/ .

The Data Engineer will help build and support a secure and privacy-first data foundation to support stakeholders across the university to serve the academic and research needs of the university. The ideal candidate will identify opportunities to acquire the skills and knowledge to support critical data needs, applying best practices. The candidate will work closely with mentors and engineering leadership, prototyping and deploying solutions, and collaboratively implementing solutions in a high-performance academic environment. Excellent Data Engineering builds on software engineering fundamentals, and the successful candidate will bring experience and knowledge on engineering practices and computer science theory, along with a curiosity for new technologies. They will play a critical role in advancing the organization’s Data Engineering and Data Science service offering by working through both the design and development of solutions as well as working hands-on in the data. This position reports to the Director of the Data and Analytics practice and works closely with data analysts and data engineering personnel.

Successful candidates will possess a foundation in engineering, with experience and expertise in managing data and data pipelines, along with a curiosity and aptitude for applied mathematics, especially with respect to machine learning and statistical inference applied in the higher educational domain.

• Develop and automate efficient, maintainable and scalable data pipelines using python and pandas
• Store and transform data in Postgres and Oracle databases to serve downstream visualization tools
• Automate builds and infrastructure deployments with GitLab CI/CD
• Develop and deploy applications to OpenShift/OKD and Azure
• Build developer tools in bash, python to increase team productivity
• Write clean python code and ensure high code quality is maintained throughout all projects
• Create clear and concise documentation
• Perform code reviews and participates in architecture and design discussions
• Experiment with new technologies that can improve maintainability, performance and quality of our data pipelines or lead to other process improvements within the team
• Develop expertise in Duke OIT data center infrastructure and cloud infrastructure domains
• Support teams applying complex reporting and analytics techniques on institutional datasets
• Learn Data Engineering tooling, processes, and workflows to improve the quality of data pipelines and data assets
• Review data requirements from stakeholder Analysts and Data Scientists and deliver prepared data for analytics and modeling
• Leverage internal and external tools and resources to improve pipeline stability and data quality

• Experience in software engineering, preferably in higher education
• Experience with large-scale data warehousing architecture and data modeling
• 1-2 years of experience hands on experience with relational databases and authoring SQL queries
• 2+ years experience in high level programming languages such as Python, Java, or Scala
• Understanding of different data formatting (JSON, CSV, Parquet) and experience with complex data structures
• Experience with systems for managing data workflows, such as Airflow, Prefect, or Dagster
• Experience deploying open-source machine learning and artificial intelligence packages in enterprise environments
• Experience working with Git, including understanding of collaborative workflows and branching strategies
• Experience working within one or more cloud providers (Azure, AWS, GCP)
• Ability to communicate and document engineering designs in a logical and understandable manner for varying audiences
• Demonstrated experience writing maintainable, production-ready software
• Passion for turning data into actionable insights to improve student experience and support research success
• Demonstrated curiosity for discovering and learning new tools and technologies, with an ability to understand new technology concepts quickly and apply them accurately through an evolving, dynamic environment
• Enjoyment working both independently and collaboratively in an agile environment
• Ability to prioritize and manage multiple projects at the same time
• Excellent communication skills that allow smooth collaboration with stakeholders
• Experience with CI/CD flows a plus

Education and Experience

Bachelor degree in computer science, information systems, or other technical field; or equivalent combination of relevant education and experience.

Duke is an Affirmative Action/Equal Opportunity Employer committed to providing employment opportunity without regard to an individual's age, color, disability, gender, gender expression, gender identity, genetic information, national origin, race, religion, sex, sexual orientation, or veteran status.

Duke aspires to create a community built on collaboration, innovation, creativity, and belonging. Our collective success depends on the robust exchange of ideas—an exchange that is best when the rich diversity of our perspectives, backgrounds, and experiences flourishes. To achieve this exchange, it is essential that all members of the community feel secure and welcome, that the contributions of all individuals are respected, and that all voices are heard. All members of our community have a responsibility to uphold these values.

Essential Physical Job Functions: Certain jobs at Duke University and Duke University Health System may include essential job functions that require specific physical and/or mental abilities. Additional information and provision for requests for reasonable accommodation will be provided by each hiring department.