Joshua Starmer PhD

This paper examines the economics and equity challenges of implementing LLMs in healthcare, focusing on three pathways: training from scratch, fine-tuning pretrained models, and using out-of-the-box models, and evaluates their trade-offs in cost, customization, and accessibility. 1️⃣ Training from scratch offers maximum customization and transparency by building LLMs on specific healthcare data but requires significant resources in infrastructure, workforce, and upfront costs. 2️⃣ Fine-tuning pretrained models balances cost and customization by adapting existing LLMs with healthcare-specific data but risks biases from the original training data and has limited transparency. 3️⃣ Using out-of-the-box models enables rapid deployment with minimal infrastructure and workforce needs but offers limited customization and transparency, often relying on general-purpose LLMs. 4️⃣ Cloud service providers play a critical role in scaling LLMs through managed services, federated learning, and hardware accelerators, but affordability and equitable access remain challenges. 5️⃣ Privacy, ethics, and regulatory considerations, including data deidentification and compliance with standards such as HIPAA and GDPR, are essential for LLM adoption in healthcare. 6️⃣ Potential applications of LLMs include chatbots, clinical decision support, and data summarization, but ongoing efforts are needed to mitigate biases, toxicity, and ensure model transparency. ✍🏻 Radha Nagarajan, FHIMSS, FAMIA, Midori Kondo, Franz Salas, Emre Sezgin, Yuan Yao, Vanessa K., Sandip Godambe, Naqi Khan, MD, MS, Alfonso Limon, Ph.D., Graham Stephenson, Sharief Taraman, MD, DABPN, DABPM, FAAP, Nephi Walton, Louis Ehwerhemuepha, Jay Pandit, Deepti Pandita, Michael Weiss, Charles Golden, Adam Gold, John Henderson, Angela Shippy MD, Leo Anthony Celi, Bill Hogan, Eric Oermann, Terence Sanger, Steven Martel, MD. Economics and Equity of Large Language Models: Health Care Perspective. Journal of Medical Internet Research. 2024. DOI: 10.2196/64226 📄 Planning to submit your research to a peer-reviewed, open-access journal? Use the discount code AMJB200 at checkout to save $200 USD on the article processing charge at any JMIR Publications journal. Start your submission here: https://lnkd.in/deuAEN2r. It's important to me that you know this isn't a sponsored post. I'm simply committed to keeping you informed about opportunities to disseminate your research effectively.

111

16 Comments

There is often an unspoken assumption by ML researchers that all we need to make progress and model every aspect of our world is data. We do not ascribe to this in my lab. In fact, we believe that data will never be enough. Our experiences with the nuances and complexities of scientific problems have informed us to the insufficiency of data to capture continuous physical processes, which afterall govern our biological and physical world. An example of this is this series of two papers led by my wonderful PhD student, Anowarul Kabir and advanced by a precious multi-year collaboration of my lab with Los Alamos National Lab: Anowarul Kabir, Manish Bhattarai, Kim Rasmussen, Amarda Shehu, Anny Usheva, Alan R Bishop, and Boian S Alexandrov. Examining DNA Breathing with pyDNA-EPBD. Bioinformatics 39(11):btad699, 2023. https://lnkd.in/gZaHE4hS Anowarul Kabir, Manish Bhattarai, Selma Peterson, Yonatan Najman-Licht, Kim Ø Rasmussen, Amarda Shehu, Alan R Bishop, Boian Alexandrov, Anny Usheva. DNA Breathing Integration with Deep learning Foundational Model Advances Genome-wide Binding Prediction of Human Transcription Factors. Nucleic Acids Research: gkae783, 2024. https://lnkd.in/gfTTqcPy Our goal: advance an exceptionally challenging problem in molecular biology, prediction of transcription factor binding sites. Our first step: capture the underlying physics that is missing in the data. Our second step: integrate that now with the data we have in a foundation model for predicting transcription factor binding sites. Performance improves. Most importantly, when we look at the sequence motifs that constitute "signatures" of what makes a transcription factor binding site, we obtain answers. All in the open, nothing opaque.

33

1 Comment

Foundational models, or large language models (LLMs), train at the scale of the entire internet to acquire the vocabulary, syntax, and narrative structure of human language. Fine-tuning involves using a smaller, domain-specific dataset to grant additional specificity to a pre-existing foundational model that has already acquired patterns and features from massive underlying training datasets. By utilizing the existing knowledge embedded in a pre-trained language model, we unlock high-performance language understanding, but fine-tuning helps to get the right answer - not just an answer that sounds right - with a minimum of hallucinations or incorrect info. In healthcare, out-of-the-box, foundational models are often too generic to sufficiently answer the precision of a healthcare query. Foundational language models are trained on a massive scale of data, which includes jokes, incorrect information, misunderstandings, and misinformation, along with the sum total of human knowledge. The problem is, if you ask a large language model to give you factually correct information when it has been trained on BOTH accurate and inaccurate information - you’re playing a bit of a dangerous game. Enter fine-tuning language models. In the old days, we used to build a single model for a single task - for example, classifying procedures into evidence-based or not as laid out by clinical practice guidelines, for patients with particular disease markers. Each disease would have its own model, which took a long time to build and didn’t scale particularly well, and we struggled with the sparsity and inherent high-dimensionality of healthcare data. By fine-tuning LLMs, however, we're able to specialize those foundational models to achieve greater domain specificity and specialization, leverage the unique predictive and planning capabilities of LLMs, and cut down on the incorrect information that such models inevitably produce, while still preserving the underlying language prowess of the LLM. If improved accuracy and specificity wasn't enough of a selling point, there's another notable upside to fine-tuning foundational models for use in a clinical context. In terms of computational efficiency, using LLMs out of the box on specific tasks can require massive compute (and spend) that amounts to computational overkill for the problem. Some LLMs charge per query, some by compute load or time - either way, as a business running LLMs in a production environment, minimizing the number of LLM calls means optimizing your compute expenditures, too. When LLMs fail at scale in healthcare, it's usually along the fault line of failure to specify adequately. Fine-tuning large language models along domain-specific lines is one way to effectively productize genAI for healthcare use cases. #genAI #LLM #ai #healthcareAI #healthtech #largelanguagemodels #generativeAI #chatGPT

61

2 Comments

Nature article highlighting the collapse of generative models as training data of future generations uses output of previous generations.    https://lnkd.in/eNzsZ6F4

10

The Information explores how OpenAI's reasoning models could help counteract potentially slowing progress from pretraining of larger more sophisticated models with more incremental gains over earlier models. It also notes that "relatively few developers" are using o1 models. While I haven't seen the full-fledged o1 models (and am certainly not a developer), the current models sometimes think hard about how to break code -- have seen them replace functional authentication with placeholders that you have to go back and update later, or swap out the correct variables with wrong ones -- for instance upper- or lowercasing something you didn't ask for, or adding an underscore where you didn't want one. At present, I think it still is better to have tighter feedback loops with the user -- otherwise, the genAI is thinking long and hard -- while a game of telephone can ensue where mistakes sometimes increase in the process of reflection -- about something that might not be what you wanted. A snippet from the paywalled article. The Information has done a lot of unique reporting on the subject.

4

1 Comment

👉🏼 Expansive data, extensive model: Investigating discussion topics around LLM through unsupervised machine learning in academic papers and news 🤓 Hae Sun Jung 👇🏻 https://lnkd.in/ef_MF54v 🔍 Focus on data insights: - Utilized data from Web of Science and LexisNexis to explore topic modeling methods for Large Language Models (LLMs). - Evaluated various topic modeling approaches such as LDA, NMF, CTM, and BERTopic based on diversity and coherence metrics. - BERTopic demonstrated superior performance in diversity and coherence across both platforms. 💡 Main outcomes and implications: - News articles covered a wide range of topics related to LLM applications, focusing on specialized domains. - Research papers concentrated on technical aspects of LLM technology. - Insights gained can help understand future challenges and opportunities for LLMs in different sectors. 📚 Field significance: - Enhances understanding of how LLMs are utilized in academic and news contexts. - Provides insights for enterprises leveraging LLMs for service delivery. 🗄️: #topicmodeling #unsupervisedlearning #machinelearning #datainsights

1

Check out this highly recommended course for those interested in optimizing their models: - Introduction to Quantization - Symmetric and Asymmetric Quantization Modes - Hands-on Implementation from Scratch using PyTorch - covering Per Tensor, Per Channel (Axis), and Per Group techniques - Weights Packing: Exploring 2 or 4 Bits Quantization Methods https://lnkd.in/gMAuWXHm

43

Synthetic Data for LLM applications Synthetic healthcare datasets have been around for some time. Whenever we discuss synthetic data, we often find ourselves trapped in one question: How realistic is the synthetic data? We create synthetic data because we cannot share the original data for various obvious reasons, e.g., privacy, security, etc. The challenging part is that we want the synthetic data to have all the nice characteristics of the original data. With all the challenges aside, many researchers adopted and tried to prove the value of synthetic data. This morning, I came across this well-summarized survey article, "Synthetic data in healthcare: A narrative review," by Aldren Gonzales, Guruprabha Guruswamy, and Scott Smith. According to the paper, people have used synthetic data for seven categories: 1. Simulation and Prediction Research 2. Hypothesis, Methods, and Algorithm Testing 3. Epidemiological Study/Public Health Research 4. Health IT Development and Testing 5. Education and Training 6. Public Release of Datasets 7. Linking data I have been thinking about synthetic data quite a bit these days. It's because of the Large Language Models (LLMs) like ChatGPT by OpenAI and Claude by Anthropic. I have been playing around with those LLMs. I realize these days that, even if you are an expert, it's almost impossible to predict the outputs. As we cannot logically bind the output behaviors, testing an LLM model with all possible(or available) cases is the only way. Here, I think, is the new application of synthetic data: LLM application testing. If we were to build many LLM applications, we would need a lot of synthetic healthcare data. In this case, the synthetic data do not need to be hyper-realistic. They just need to represent the "quirkiness" of the real-life use cases, so weird synthetic data should be fine. Healthcare-focused LLM applications would need to test with all sorts of available synthetic data to see if such applications produce weird outputs. We may need a mechanism to do so. I think this new use case of synthetic data will be critical in healthcare. Let's see. [1] https://lnkd.in/e8DiEH9j

64

9 Comments

FOCUSING ML HEALTHCARE RESEARCH ON VERIFIABLE, EQUITABLE PATIENT BENEFITS ⬇️ This paper focuses on shifting the application of ML in healthcare from mere technical novelty to achieving equitable impact. It critically examines how the current incentive structures in healthcare ML research prioritize technological innovation over real-world clinical outcomes, and proposes a reorientation towards impactful applications. 1️⃣ Emphasizes the need to distinguish between ML applied to healthcare data for technical challenges and ML designed with the primary goal of solving clinical problems. 2️⃣ Highlights the common disconnect between healthcare ML research goals and tangible clinical impacts, suggesting that it should aim for direct improvements in patient care. 3️⃣ Critiques the healthcare ML community for often neglecting meaningful deployment and integration of technologies within healthcare settings, leading to limited practical use despite technical advancements. 4️⃣ Proposes a series of "Impact Challenges" designed to refocus efforts on real-world health equity and effective clinical applications of ML technologies. 5️⃣ Discusses structural barriers within the field, such as funding biases towards novelty, and offers recommendations for promoting research that addresses these barriers to enhance the real-world applicability and impact of healthcare ML. ✍🏻 Aparna Balagopalan, Ioana Baldini, Leo Anthony Celi, Judy Gichoya Liam McCoy, Tristan Naumann, Uri Shalit, Mihaela van der Schaar, Kiri L. Wagstaff. PLOS Digital Health. April 15, 2024. DOI: 10.1371/journal.pdig.0000474 💡 Focusing on equitable impact is not just ethically imperative but also essential for the sustainable integration of AI technologies in healthcare, where they can genuinely contribute to health outcomes and not exacerbate existing disparities. 💥 Join our community of 30.000 curious minds! ✅ Subscribe to our newsletter for your front-row seat to the latest groundbreaking studies. Get started here: https://lnkd.in/eR7qichj.

144

7 Comments

Conditional Computation in LLMs (MoEs): Understanding the Concept of MoEs The Mixture of Experts (MoE) works on the idea of conditional computation, allowing the model to only run through some parts (an Expert) for a given input compared to a dense model where all parameters are used. It consists of a gating mechanism that selects the experts for the given input, enabling the model to scale without increasing the required computation. Popular LLM architectures like Mixtral and Grok use the same concept. What is MoEs? The Mixture of Experts (MoE) makes a simple modification to the usual architecture by replacing all or some of the Feed Forward layers with an MoE layer. The MoE layer consists of: 1. Router: A parameter gating mechanism that selects the experts used to process the given input by producing a probability distribution over experts. 2. Experts: Standalone NN modules with independent sets of parameters trained jointly with routers. It can use more complex architecture to create hierarchical MoE modules by implementing each expert as another MoE. - Implementation in Switch Transformers : Training was found to be 7x faster than training the base T5, keeping computation and training time constant. - Gshard and MeshTransformer : These frameworks propose efficient methods to train the MoE architectures using the parallelisation of models across GPUs/TPUs. Challenges with MoEs and Solutions 1. Inconsistent Batch Size for Experts: Can be overcome with the use of "soft" constraints in training loss. 2. Load Balancing Across Experts: The issue of repeatedly using the same few experts while training can be overcome by including better balancing mechanisms. 3. Model Instability Issues: Use of router-z Loss as an auxiliary loss to encourage smaller values within the router mechanism. 4. Model Overfitting: Requires different hyperparameter settings for effective fine-tuning. It was found that models benefit from high learning rates and smaller batch sizes. Recommended Reading/References 1. Hugging Face Blog on MoE - https://lnkd.in/g36gQJTx 2. Cameron Wolfe's Substack on Conditional Computation - https://lnkd.in/gtpjMHnJ

60

1 Comment

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

29

At a time when the hype cycle for “AI” is in various stages of peaking, the awarding of the Nobel in Physics to Hopfield and Hinton for their work on #MachineLearning is a good time to reflect on the impacts we already experience from this field: 🔬Diagnostic imaging accuracy, speed, and ability to find life threatening conditions 💊 Drug development and new uses for existing treatments that treat what was previously incurable 💸 Fraud detection, credit scoring, and automated investing make financial tools and resources more accessible to more people in need 🚗 Public transportation and self-driving vehicles make driving safer and less impactful to the environment 🛍️ Retail and marketing in general are more personalized and real-time 📱Technology like speech recognition and translation allow us to communicate, learn, and share more seamlessly and globally ➡️➡️What are some of the ways you experience the impact of ML models in life and work? ⬅️⬅️ #AI #ml #cybersecurity #nobelprize #nobel2024 #artificialintelligence #healthcare #research #drugdiscovery #imaging #predictivemedicine #medicine #surgery #radiology #pharmacy #predictiveanalytics #neuralnetworks #gpu #medtech #fintech #edtech Siemens Healthineers GE HealthCare Roche Novartis Pfizer Moderna Medtronic Intuitive Merck AstraZeneca Tesla Apple Amazon Amazon Web Services (AWS) Google OpenAI Microsoft Toshiba Philips NVIDIA AMD TSMC https://lnkd.in/eePQtdaY

15

1 Comment

Snowflake’s BUILD 2024 AI/ML Development Conference Highlight – AI, Agents, and Data by Andrew Ng (Founder of LandingAI) The session centered on the evolution of AI with a specific focus on AI agents and the emerging paradigm of agentic reasoning. This new approach emphasizes the ability of AI to engage in iterative, goal-directed workflows, fundamentally transforming how applications are developed and deployed. With the exponential growth in the volume and complexity of data, the role of unstructured data—text, images, audio, and video—is becoming a cornerstone in driving automation and innovation across industries. Ng's discussion explored how this convergence of AI agents, agentic reasoning, and unstructured data is poised to revolutionize automation and application development in a cost-effective and scalable way. Recent advancements in large language models (LLMs) and large multimodal models (LMMs) have drawn significant attention due to their broad applicability and power. Building on these foundational technologies, AI agents are emerging as the next significant breakthrough. These agents, designed to reason and interact iteratively, are proving transformative in industries that require complex decision-making and operational efficiencies. As the emphasis shifts from models to applications, the importance of effective data engineering and the integration of unstructured data in workflows will shape the next generation of AI-driven solutions. Video Link: https://lnkd.in/e2njUdFC #ArtificialIntelligence #AIAgents #AgenticReasoning #LLMs #LargeLanguageModels #Innovation #LargeMultimodalModels #LMMs

9

Looking how to select better representatives from your clusters? Take a look at Lexin Chen's PRIME repo: https://lnkd.in/enM_Btbx Now with updated functionality and tutorials! Great collaboration with Arup Mondal and Alberto Perez! #moleculardynamics #clustering #machinelearning

65