Applying Large Language Models (LLM) in Drug Safety Reviews Fellowship

*Applications will be reviewed on a rolling basis

This project is in the Office of Pharmacovigilance and Epidemiology (OPE), Office of Surveillance and Epidemiology (OSE). This project will evaluate the performance of LLMs in selected literature review tasks (i.e., literature screening, information extraction). Collect sample literature reviews completed by DEPI reviewers and summarize these reviews to generate testing datasets for LLM evaluation. Research the prompt engineering techniques and LLM performance metrics. The performance of LLMs will be analyzed against the gold standards generated by FDA reviewers. This project will be conducted in collaboration with AI subject matter experts at NCTR/FDA. 

Under the guidance of the mentor, the participant will gain in-depth knowledge about the pharmacoepidemiology literature review for drug safety evaluation, with a focus on literature screening and information extraction. The participant will learn how to apply the state-of-the-art LLMs (e.g., GPT, Llama 2) in selected tasks of literature review, and will receive hands-on experience of evaluating the performance of LLMs in these tasks. This is a unique opportunity for candidates to gain first-handed knowledge and experience of using LLMs for enhanced drug safety review.

The Center for Drug Evaluation and Research (CDER) performs an essential public health task by making sure that safe and effective drugs are available to improve the health of people in the United States. As part of the U.S. Food and Drug Administration (FDA), CDER regulates over-the-counter and prescription drugs, including biological therapeutics and generic drugs. This work covers more than just medicines.

This program, administered by ORAU through its contract with the U.S. Department of Energy to manage the Oak Ridge Institute for Science and Education, was established through an interagency agreement between DOE and FDA. The initial appointment is for one year, but may be renewed upon recommendation of FDA contingent on the availability of funds. The participant will receive a monthly stipend commensurate with educational level and experience. Proof of health insurance is required for participation in this program. The appointment is full-time at FDA in the Silver Spring, Maryland, area. Participants do not become employees of FDA, DOE or the program administrator, and there are no employment-related benefits.

Completion of a successful background investigation by the Office of Personnel Management is required for an applicant to be on-boarded at FDA.  OPM can complete a background investigation only for individuals, including non-US Citizens, who have resided in the US for a total of three of the past five years.

FDA Ethics Requirements

If an ORISE Fellow, to include their spouse and minor children, reports what is identified as a Significantly Regulated Organization (SRO) or prohibited investment fund financial interest in any amount, or a relationship with an SRO, except for spousal employment with an SRO, and the individual will not voluntarily divest the financial interest or terminate the relationship, then the individual is not placed at FDA.   For additional requirements, see FDA Ethics for Nonemployee Scientists. 

FDA requires ORISE participants to read and sign their FDA Education and Training Agreement within 30 days of his/her start date, setting forth the conditions and expectations for his/her educational appointment at the agency.  This agreement covers such topics as the following:

• Non-employee nature of the ORISE appointment;
• Prohibition on ORISE Fellows performing inherently governmental functions;
• Obligation of ORISE Fellows to convey all necessary rights to the FDA regarding intellectual property conceived or first reduced to practice during their fellowship;
• The fact that research materials and laboratory notebooks are the property of the FDA;
• ORISE fellow’s obligation to protect and not to further disclose or use non-public information.