Activity 2: Institutional Complexities in AI Deployment

Introduction

Your laboratory has decided it wants to implement a machine learning solution for detecting IV fluid contamination in BMP results. This will require engagement and alignment across several stakeholders across the organization: Lab Operations, IT/IS, Data Science, Compliance, and Providers & Patients. Each of these stakeholders have their own priorities, incentives, and responsibilities.

For this activity, we will simulate the discussions that would unfold as the implementation efforts would unfold. Each participant will be assigned a group to represent, providing their role-specific guidance to the overall efforts. Your group leader will be available for hints and suggestions at any time.

Primary Responsibility

Ensure that AI applications improve laboratory quality, workflow efficiency, and/or patient outcomes without compromising clinical standards.

Secondary Responsibilities
  • Serve as liaison between laboratory staff, clinicians, and technical teams.
  • Provide clinical input on algorithm development and use case prioritization.
Deal Breakers
  • Insufficient validation or failure to meet clinical performance benchmarks.
  • Disruption to laboratory operations or patient care.
  • Lack of clinician trust or engagement.

1) What should we consider when designing a validation study?

2) What metrics should be used to gauge performance?

3) How can we approach defining “acceptable performance”?

1) Will this process be fully automated or have a human in the loop?

2) Is the data infrastructure ready to support this use case?

3) How will infrastructure needs be prioritized and paid for?

1) How do we monitor performance over time?

2) Who is responsible for this monitoring?

3) What protocols need to be developed for downtime and change control?

Primary Responsibility

Ensure secure, reliable, and scalable deployment, integration, and ongoing maintenance of AI systems within the existing institutional IT infrastructure.

Secondary Responsibilities
  • Maintain cybersecurity and data privacy.
  • Provide technical support and incident response.
  • Oversee interoperability with laboratory and hospital information systems.
Deal Breakers
  • Security vulnerabilities or non-compliance with institutional IT policies.
  • Lack of interoperability or inability to integrate with existing systems.
  • Excessive resource requirements or unsustainable maintenance burden.

1) What should we consider when designing a validation study?

2) What metrics should be used to gauge performance?

3) How can we approach defining “acceptable performance”?

1) Will this process be fully automated or have a human in the loop?

2) Is the data infrastructure ready to support this use case?

3) How will infrastructure needs be prioritized and paid for?

1) How do we monitor performance over time?

2) Who is responsible for this monitoring?

3) What protocols need to be developed for downtime and change control?

Primary Responsibility

Develop, validate, and monitor AI models to ensure accuracy, fairness, and relevance to clinical needs.

Secondary Responsibilities
  • Collaborate with clinicians to define use cases and performance metrics.
  • Oversee validation, verification, and ongoing clinical performance monitoring of AI systems.
  • Provide transparency regarding model limitations and potential biases.
Deal Breakers
  • Insufficient, biased, or unrepresentative data.
  • Lack of stakeholder engagement or infrastructure support.

1) What should we consider when designing a validation study?

2) What metrics should be used to gauge performance?

3) How can we approach defining “acceptable performance”?

1) Will this process be fully automated or have a human in the loop?

2) Is the data infrastructure ready to support this use case?

3) How will infrastructure needs be prioritized and paid for?

1) How do we monitor performance over time?

2) Who is responsible for this monitoring?

3) What protocols need to be developed for downtime and change control?

Primary Responsibility

Ensure full compliance with all applicable regulatory, legal, and accreditation requirements (e.g., CLIA, CAP, HIPAA, FDA).

Secondary Responsibilities
  • Guide risk assessments and documentation practices.
  • Monitor for ongoing regulatory changes impacting AI use.
  • Oversee management of adverse events and reporting obligations.
Deal Breakers
  • Non-compliance with regulatory or accreditation standards.
  • Insufficient documentation or inability to audit system performance and decision-making.

1) What should we consider when designing a validation study?

2) What metrics should be used to gauge performance?

3) How can we approach defining “acceptable performance”?

1) Will this process be fully automated or have a human in the loop?

2) Is the data infrastructure ready to support this use case?

3) How will infrastructure needs be prioritized and paid for?

1) How do we monitor performance over time?

2) Who is responsible for this monitoring?

3) What protocols need to be developed for downtime and change control?

Primary Responsibility

Ensure that AI tools support and enhance patient care and improve provider or patient experience.

Secondary Responsibilities
  • Participate in user feedback and acceptability assessments.
  • Identify unintended consequences or workflow disruptions.
  • Provide input on clinical relevance and user experience.
Deal Breakers
  • Loss of provider autonomy or increased workload.
  • Lack of trust or understanding regarding AI system recommendations.

1) What should we consider when designing a validation study?

2) What metrics should be used to gauge performance?

3) How can we approach defining “acceptable performance”?

1) Will this process be fully automated or have a human in the loop?

2) Is the data infrastructure ready to support this use case?

3) How will infrastructure needs be prioritized and paid for?

1) How do we monitor performance over time?

2) Who is responsible for this monitoring?

3) What protocols need to be developed for downtime and change control?