Short answer
The feature was an AI-assisted patient intake inside a multi-branch clinic's management system: a guided conversation that collects symptoms and history, with a language model turning the responses into a structured summary for attending staff. It does not diagnose. It summarizes and flags, and every summary goes to a front desk staff member who can edit, reject, or approve it before it touches the patient record. What nearly went wrong was patient language, because real patients describe symptoms colloquially, sometimes in Tagalog, sometimes switching languages mid-sentence, and the launch slipped three weeks while we rebuilt pre-processing and prompts to handle it.
The first time you integrate AI into a production healthcare product, you discover very quickly that the margin for error is different from a typical web application. This is the account of our first AI feature in a clinic management system, what we got right, what nearly went wrong, and what we now do differently as a result.
What We Built and Why
The client was a multi-branch clinic running a proprietary patient management system we had built for them the year prior. The intake process - new patient registration, symptom collection, triage priority assignment - was handled by a combination of front desk staff and paper forms, then entered into the system manually.
The brief was to reduce the manual data entry load on front desk staff and flag high-priority patients faster. We scoped an AI-assisted intake: a structured conversation that collects symptoms and patient history through a guided interface, with a language model processing the responses and generating a structured intake summary for the attending staff.
The feature does not diagnose. It summarizes and flags. That distinction was fundamental to everything that followed.
What We Got Right
The most important early decision was defining what the AI could and could not say. We wrote an explicit constraint set before we wrote a single prompt: the system could summarize what the patient reported, it could flag keywords associated with high-priority conditions according to the clinic's own triage criteria, and it could generate structured data for staff review. It could not make diagnostic statements, suggest medications, or present any output as clinical assessment.
This sounds like obvious caution. In practice, it required pushing back on some initial scope ideas from the client who wanted the system to "tell staff what was probably wrong." That framing was not appropriate for this feature, and we had to explain why clearly enough that the client agreed to reframe it.
The second good decision was keeping a human review step mandatory. Every AI-generated intake summary went to a front desk staff member before it touched the patient record. The staff member could edit, reject, or approve the summary. The AI was a drafting tool, not an autonomous input.
What Almost Broke Us
We underestimated the variability of patient language. Our initial prompts were tuned on structured symptom descriptions. Real patients describe symptoms in ways that are colloquial, sometimes vague, sometimes in Tagalog, sometimes mixing languages mid-sentence. The model's outputs on highly unstructured input were significantly worse than on our test cases.
The fix required a better pre-processing step and more careful prompt engineering than we had initially scoped. We had not budgeted sufficient time for this, and the feature launch was delayed by three weeks while we worked through it.
We also had one incident during testing where the model generated a summary that technically fit within our constraints but used phrasing that the clinic's medical director felt was too clinical in tone - implying more certainty than a patient summary should carry. We revised the output formatting to make the provisional nature of the summary explicit in every output.
What We Do Differently Now
When we scope AI solutions for healthcare or any high-stakes workflow, we now include an explicit "red line" session in discovery. We work with the client to define the exact outputs that are acceptable and the exact outputs that are not, before we write a prompt. That session creates the constraint document that guides prompt engineering and testing.
We also scope language variability as a first-class concern, not a secondary edge case. If a product will be used by people who do not communicate in formal, structured language, the AI feature needs to be tested against that full range of input before it goes live.
The feature is working well now. Front desk processing time dropped measurably and high-priority cases are flagged more consistently. It took longer to get there than we planned, but it is reliable.
If you are thinking about an AI integration for a clinical or operational workflow, the most important conversation is about constraints and error modes before it is about capabilities.
Questions · 05
Frequently asked questions
How do you make an AI feature safe in a clinical workflow?
By writing an explicit constraint set before writing a single prompt. On this build the system could summarize what the patient reported, flag keywords associated with high-priority conditions according to the clinic's own triage criteria, and generate structured data for staff review. It could not make diagnostic statements, suggest medications, or present any output as clinical assessment. The margin for error in a production healthcare product is different from a typical web application, and that difference has to be encoded in scope.
What did the client originally ask for that you pushed back on?
The client wanted the system to tell staff what was probably wrong. That framing was not appropriate for this feature, and we had to explain why clearly enough that the client agreed to reframe it. Pushing back on scope in a high-stakes workflow is part of the job, not an obstacle to it.
What is a red line session?
An explicit discovery session where we work with the client to define the exact outputs that are acceptable and the exact outputs that are not, before anyone writes a prompt. It produces a constraint document that then guides prompt engineering and testing. We now include one when scoping AI solutions for healthcare or any other high-stakes workflow.
How do you handle Tagalog and mixed-language user input?
We scope language variability as a first-class concern rather than a secondary edge case. If a product will be used by people who do not communicate in formal, structured language, the AI feature has to be tested against that full range of input before it goes live. On this build we underestimated it, and prompts tuned on structured symptom descriptions performed significantly worse on real, unstructured patient language.
Did the feature work in the end?
It is working well now. Front desk processing time dropped measurably and high-priority cases are flagged more consistently. We also revised the output formatting during testing after the clinic's medical director found one summary too clinical in tone, so every output now makes its provisional nature explicit.
