Activity 3: Institutional Complexities in AI Deployment

Role-Playing an AI Implementation

Your laboratory is evaluating two AI-enabled projects that could change how laboratory results are reviewed, released, communicated, and acted on:

  1. Part 1: Considering and implementing a vendor autoverification solution that automates result verification using analyzer flags, quality checks, delta checks, reference intervals, clinical context, and model-derived risk scores.

  2. Part 2: Implementing a generative AI solution that drafts interpretive comments, internal review notes, or provider-facing explanations after results are held, corrected, or require additional context.

Each participant will represent one institutional perspective. Your goal is not to “win” the discussion, but to identify what each group needs before the project can move forward, what risks must be controlled, and where institutional responsibilities overlap.

Shared Questions
  • What problem is the AI system solving, and who is accountable if it fails?
  • What evidence is needed before pilot use, clinical use, and broader deployment?
  • Where should human review be required?
  • What needs to be documented for validation, governance, downtime, monitoring, and change control?
  • What risks are different for a vendor autoverification tool versus a locally configured generative AI tool?
Primary Responsibility

Ensure the AI workflow improves result release, turnaround time, and operational reliability without compromising review of abnormal, questionable, or high-risk results.

Key Concerns
  • Manual review burden, staffing impact, and escalation pathways.
  • Fit with existing autoverification rules, delta checks, specimen rejection criteria, and result-release workflows.
  • Clear ownership for reviewing held results, documenting overrides, and releasing results.
Deal Breakers
  • Inappropriate release of results that should have been manually reviewed.
  • Excessive false holds that delay testing or overwhelm staff.
  • Unclear instructions for technologists during downtime, overrides, or disputed holds.
  • Output that conflicts with laboratory standard operating procedures.

1) Which result types, analytes, patient settings, and abnormal patterns should be eligible for automated release?

2) What local validation cases are needed before the laboratory trusts vendor performance claims?

3) What documentation is required when staff override a hold, manually release a result, or disable autoverification?

1) What kinds of result comments, review notes, or provider explanations are appropriate for AI drafting, and which should remain fully manual?

2) Who must review and approve generated text before it becomes part of the laboratory record or patient chart?

3) How will staff detect omissions, overstatements, or recommendations that conflict with local policy?

Primary Responsibility

Ensure the AI systems can be deployed, integrated, secured, supported, and retired within institutional infrastructure and cybersecurity standards.

Key Concerns
  • LIS, middleware, analyzer, EHR, data warehouse, and identity-management integration.
  • Vendor access, audit logging, service-level expectations, and incident response.
  • Data flows, PHI handling, retention, backup, and downtime procedures.
Deal Breakers
  • Unapproved external data transmission or insufficient security review.
  • No workable integration path with existing LIS/EHR workflows.
  • Support requirements that exceed available staffing or infrastructure.

1) What result, flag, demographic, order, medication, diagnosis, and historical data does the vendor need, and where will those data be processed?

2) How will autoverification decisions, result holds, overrides, and audit trails move between middleware, LIS, and EHR-facing workflows?

3) What uptime, support, audit-log, and version-notification requirements must be in the contract?

1) Can the generative AI tool run in an approved environment with appropriate PHI controls?

2) How will prompts, retrieved reference material, generated output, reviewer edits, and final text be logged?

3) What technical controls prevent unapproved model use, prompt injection, data leakage, or silent model changes?

Primary Responsibility

Ensure the implementation satisfies applicable CLIA, CAP, HIPAA, institutional governance, contractual, and, where relevant, FDA expectations.

Key Concerns
  • Intended use, validation scope, risk classification, and documentation quality.
  • Auditability of inputs, outputs, human review, approvals, and corrective actions.
  • Vendor claims, model updates, local modifications, and post-deployment monitoring.
Deal Breakers
  • No documented intended use or validation plan.
  • Inability to reconstruct why an alert, recommendation, or generated comment was produced.
  • Unclear responsibility for adverse events, complaints, or regulatory reporting.

1) Is this being used only to automate laboratory result release, or does it create clinical decision support obligations?

2) What evidence is needed to verify local safety, false-release risk, false-hold burden, and performance after vendor updates?

3) What records must be retained for validation, released results, held results, overrides, rule/model changes, and monitoring?

1) Is generated text part of the medical record, a draft working note, or an internal quality document?

2) What review, attestation, and version-control requirements apply before generated text is released?

3) How will the institution manage hallucinations, biased language, inappropriate recommendations, and patient privacy risks?

Primary Responsibility

Ensure AI-supported laboratory workflows improve clinical decision-making without creating confusing, delayed, or low-trust communication.

Key Concerns
  • Clarity, actionability, and timing of held, delayed, corrected, or interpretive results.
  • Avoiding alert fatigue, ambiguous recommendations, and unnecessary follow-up testing.
  • Preserving clinician trust in laboratory expertise.
Deal Breakers
  • Holds, delays, or comments that create uncertainty without actionable guidance.
  • Delayed critical results or clinically important follow-up.
  • AI-generated language that appears more definitive than the evidence supports.

1) When should providers be notified that a result was held, manually reviewed, corrected, or not eligible for autoverification?

2) What information is needed to interpret a delayed, corrected, suppressed, or manually reviewed result?

3) How should provider feedback be captured when AI-driven actions affect clinical care?

1) What makes an AI-drafted comment clinically useful rather than merely plausible?

2) How should uncertainty, limitations, and recommended follow-up be phrased?

3) How will providers know whether the final comment was reviewed and approved by laboratory professionals?

Primary Responsibility

Represent patient interests in safety, privacy, transparency, equitable performance, and understandable communication.

Key Concerns
  • Whether AI use improves safety and turnaround time without delaying care or increasing confusion.
  • Protection of sensitive laboratory and clinical data.
  • Equitable performance across patient populations and care settings.
Deal Breakers
  • Unexplained use of patient data outside expected care or quality-improvement boundaries.
  • AI-driven releases, delays, errors, or communication that patients cannot challenge or understand.
  • Monitoring plans that ignore differential performance across patient groups.

1) What patient benefit justifies using historical and real-time laboratory data to automate result release?

2) How will the institution check whether the system performs differently across patient populations?

3) What should patients be told if an AI-assisted process releases, delays, corrects, or questions a result?

1) Could generated comments be misunderstood by patients viewing results in a portal?

2) What privacy protections apply when patient data are used to generate or revise text?

3) How will patients or caregivers ask questions or correct misunderstandings caused by AI-supported communication?

Primary Responsibility

Evaluate model fitness, local performance, failure modes, monitoring design, and the technical limits of both autoverification and generative AI systems.

Key Concerns
  • Ground truth definition, validation data representativeness, and drift monitoring.
  • Model update governance, reproducibility, and performance dashboards.
  • Prompt design, retrieval quality, guardrails, human feedback, and output evaluation for generative AI.
Deal Breakers
  • Vendor autoverification performance cannot be independently verified on local data.
  • No access to sufficient metadata, version information, or output logs for monitoring.
  • Generative AI output cannot be systematically evaluated for factuality, completeness, and policy adherence.

1) What local data are needed to validate false-release risk, false-hold burden, manual review reduction, turnaround time, and subgroup performance?

2) How will the team monitor drift, subgroup performance, vendor updates, and unexpected changes in hold or release rates over time?

3) What should trigger retraining, recalibration, rollback, or suspension of the tool?

1) What test set, rubric, and review process will evaluate generated text before deployment?

2) How will prompts, retrieved documents, model versions, and reviewer edits be controlled and monitored?

3) What guardrails are needed to prevent unsupported recommendations, fabricated facts, and inappropriate tone?

Final Discussion

After each group has discussed both parts, compare the implementation plans:

  • Which risks are shared across both AI projects?
  • Which risks are unique to vendor autoverification?
  • Which risks are unique to generative AI?
  • What governance structure would allow the institution to approve, monitor, revise, or stop either tool?