The project aims to evaluate and benchmark various large language models (LLMs) for their effectiveness in specialized medical use cases. With the growing reliance on AI in healthcare, it is crucial to understand how different LLMs perform in terms of accuracy, efficiency, and cost-effectiveness. The focus will be on both closed-source models like Gemini, Anthropic, and OpenAI, as well as open-source models such as Llama and Mistral. The team will conduct a series of tests to compare these models, specifically monitoring token usage and associated costs. This project will provide valuable insights into which LLMs are best suited for medical applications, helping medX Smart Solutions make informed decisions about AI integration.

- Evaluate the performance of different LLMs in medical contexts.

- Compare closed-source and open-source models.

- Monitor token usage and cost implications.

- Provide recommendations based on findings.

The deliverables for this project will include a comprehensive report detailing the performance of each LLM in medical use cases. The report will include comparative analyses of accuracy, efficiency, and cost metrics. Additionally, the team will provide a set of recommendations for medX Smart Solutions on the most suitable LLMs for their specific needs. A presentation summarizing the findings and recommendations will also be prepared for stakeholders.

- Comprehensive performance report.

- Comparative analysis of LLMs.

- Recommendations for LLM selection.

- Presentation of findings and recommendations.