IT Consultant (Cloud Data Engineer)

At Duke Health, we're driven by a commitment to compassionate care that changes the lives of patients, their loved ones, and the greater community. No matter where your talents lie, join us and discover how we can advance health together.

About Duke Health Technology Solutions

Pursue your passion for caring and innovation with Duke Heath Technology Solutions, which is dedicated to the transformation, development, and management of enterprise information technology solutions across Duke Health. By harnessing the power of innovative technologies like cloud computing and artificial intelligence — and pairing them with a forward-thinking approach — Duke Health Technology Solutions is revolutionizing the future of health care at Duke Health and beyond.

Primary Role
As a Cloud Data Engineer, you will play a crucial role in designing, implementing, and maintaining data solutions in Duke’s Azure Cloud computing environment as well as support API development, leveraging both on-premise and the cloud data solutions.  This role will collaborate with cross-functional teams to ensure the successful delivery of data projects and contribute to the continuous improvement of our data infrastructure.  As a member of the Duke Health Technology Solutions (DHTS)-Analytics Center of Excellence (ACE), this role will also provide administrative support for tools leveraged by the ACE Cloud Data Team.

Essential Tasks/Responsibilities
1.    Data Architecture Design: Design and implement scalable, high-performance data architecture on Duke’s Azure Cloud.  Define data models, schemas, and pipelines to support the organization's data needs
2.    Data Integration: Collaborate with Cloud ELT Developers to ensure the quality and consistency of data from various sources migrated to the cloud data environment.
3.    API Development: Design, develop, and maintain APIs for accessing and manipulating data stored in cloud environments.  Implements APIs to enable seamless data access for internal and external stakeholders
4.    Data Warehousing: Design and manage cloud-based data structures for storing structured and unstructured data.
5.    Data Streaming: Implement and support real-time data solutions on cloud platforms.  Design and deploy streaming pipelines for real-time analytics and monitoring..
6.    Data Lake Development: Architect and build data lakes on the cloud platform to store and analyze large volumes of raw and unstructured data.  Implement solutions for data ingestion, storage, and processing in data lake environments
7.    Data Governance and Security: Implement the data governance processes and security measures to ensure compliance with regulations and protect sensitive data in the cloud environment.  Monitor data access and usage to maintain data privacy and security standards.
8. Monitoring and Optimization: Monitor the performance and health of cloud data infrastructure.   Identify opportunities for optimization and implement solutions to improve scalability, reliability, and cost-effectiveness
9.  Collaboration and Documentation: Collaborate with cross-functional teams to understand data requirements and deliver solutions.  Document data architecture, processes, and configuration for knowledge sharing and future reference.
10. Continuous Learning and Innovation:  Stay abreast of the latest trends and technologies in cloud computing, data engineering, and API development.  Provide input into the evaluation of new tools and technologies

Education/training
Bachelor's degree in a related field, or four years of equivalent technical experience required
Required Experience
•    Proven experience as a Cloud Data Engineer leveraging data in cloud environments and API development
•    Experience with data integration, ETL/ELT process, and data pipeline in the cloud environment
•    Proficiency in programming languages such as Python, SQLk Java, Scala, or C# for data processing and scripting
•    Experience with API development frameworks and technologies
•    Knowledge of SQL and database technologies (ie: PostgreSQL, MySQL, etc.)
•    Solid understanding of data governance, security, and compliance standards in the cloud environment
•    Experience with cloud monitoring and management tools
Preferred Experience
•    Excellent problem-solving and communication skills, with the ability to convey technical concepts to healthcare and research stakeholders.
•    Experience working with healthcare professionals and researchers to understand their healthcare and research objectives
•    Knowledge of containerization and orchestration tools such as Docker and Kubernetes in Azure environments
•    Strong proficiency in Microsoft Azure services, particularly Azure Fabric, Azure Data Factory, Azure Databrick, and Azure Service Fabric
•    Experience with data visualization tools and techniques for cloud-based environments

The information above describes the general nature and level of work assigned to this position. It is not intended to be an exhaustive list of all duties and responsibilities required of position incumbents.

Duke is an Affirmative Action/Equal Opportunity Employer committed to providing employment opportunity without regard to an individual's age, color, disability, gender, gender expression, gender identity, genetic information, national origin, race, religion, sex, sexual orientation, or veteran status.

Duke aspires to create a community built on collaboration, innovation, creativity, and belonging. Our collective success depends on the robust exchange of ideas—an exchange that is best when the rich diversity of our perspectives, backgrounds, and experiences flourishes. To achieve this exchange, it is essential that all members of the community feel secure and welcome, that the contributions of all individuals are respected, and that all voices are heard. All members of our community have a responsibility to uphold these values.

Essential Physical Job Functions: Certain jobs at Duke University and Duke University Health System may include essentialjob functions that require specific physical and/or mental abilities. Additional information and provision for requests for reasonable accommodation will be provided by each hiring department.