Evidence-Grounded Agents Improve Musculoskeletal Care Pathways

Musculoskeletal care is a longitudinal coordination problem more than a single diagnostic prediction problem. Orthopaedic teams have to connect imaging, laboratory tests, pathology, orders, operative decisions, discharge planning, and rehabilitation goals across weeks or months. The paper introduces OrthoPilot as an evidence-grounded agent for that whole pathway: it retrieves patient-specific hospital evidence and external medical knowledge, then produces structured reasoning and care outputs for sequential clinical tasks.

Figure 1 gives the clearest view of the intended scope: the system is not framed as a radiology classifier or a one-shot triage assistant, but as an agent spanning admission, diagnosis, perioperative planning, surgery, discharge, and rehabilitation.

Core Contribution

The main contribution is the coupling of a full-pathway clinical agent with a large evaluation program built around musculoskeletal care. OrthoPilot has two evidence sources. The in-hospital world contains changing patient data such as imaging, laboratory results, pathology, and medical orders. The external knowledge world supplies current medical knowledge. The system uses both to generate a reasoning trace and structured clinical outputs, which makes the claim narrower and more testable than a generic medical-chatbot claim.

The other contribution is measurement. The authors build OrthoBench from real-world electronic health records across 1,000 disease codes and 11 tasks, then validate open-ended outputs with ORACLE, a rubric-based evaluator aligned against physician rankings. That matters because many clinical LLM papers stop at question answering; this one evaluates diagnostic reasoning and management decisions across a care chain.

Technical Approach

OrthoPilot is presented as an evidence-grounded agent rather than a standalone foundation model. The architecture retrieves case-specific records from hospital systems and condition-specific information from external sources before generating responses. The output is not only an answer, but a trace of the evidence used to support it. That design targets a real failure mode in longitudinal care: evidence fragmentation across departments and visits.

Figure 2 shows how the benchmark and evaluator are built. OrthoBench covers rare and non-rare musculoskeletal disease codes, while ORACLE scores open-ended clinical-management answers against specialist rubrics. The validation plot compares ORACLE scores with physician rankings across seven management tasks, which is an important check because automated grading in clinical settings can otherwise become a hidden source of bias.

Results and Analysis

The evaluation is unusually broad for a clinical-agent paper. Retrospectively, the authors compare 22 systems on OrthoBench, including reasoning LLMs, general-purpose LLMs, medical-domain LLMs, agent methods, CHEESE, and OrthoPilot. They also test external generalisation across 60 hospitals and 6,796 cases against three locally deployable open-source baselines: DeepSeek-R1, Qwen3-235B-A22B, and MedGemma-27B. The figure reports OrthoPilot ahead of the evaluated systems at the aggregate level and across task-level diagnostic and management settings.

Figure 3 is the central evidence figure because it combines benchmark ranking, external-centre validation, sequential diagnostic stages, and seven open-ended management tasks. The pattern supports the paper's main claim that evidence retrieval plus pathway-specific orchestration helps beyond simply scaling a general LLM.

The prospective studies are the stronger part of the paper. In a crossover reader study of 1,870 complex cases, OrthoPilot improved full-chain management success by 10.6%. In an 8-month randomized deployment at Ruijin Hospital with 8,240 inpatients, integration into routine care increased cumulative cases per bed by 9.7% and improved patient-reported access to health information. Those are operational endpoints, not just benchmark scores.

The interpretation should still be cautious. The deployment evidence shows that OrthoPilot can improve decision workflows and throughput in one hospital environment, while the external validation shows performance across many centres retrospectively. It does not by itself prove broad improvement in long-term functional recovery, complication rates, or rehabilitation outcomes. The paper is strongest as evidence that grounding an LLM agent in live hospital data and specialist knowledge can improve musculoskeletal-care decisions at scale; the harder clinical-outcome claim remains only partially answered.

Limitations

The main limitation is endpoint selection. The randomized deployment reports workflow and patient-information-access gains, but the abstracted results do not establish durable improvements in pain, mobility, revision surgery, or long-term rehabilitation. The deployment also took place at Ruijin Hospital, so implementation quality, hospital data integration, and clinician adoption may affect transfer to other systems. ORACLE is physician-validated, but it is still an automated evaluator for open-ended management tasks, so the clinical meaning of score differences depends on the rubric design.