Welcome to Real Talk on AI and I care with your host, Dr. Scott Morris. And I'm co host, Dr. Ray Han Ahmed.
In each episode, we discuss current news, adding a little AI education, and debate innovative AI tools and topics that will change the industry. Hello, everybody. Welcome to Real Talk. This is your co host, Scott Morris.
And I am your other co host, Ray Han Ahmed. Well, Saran, today, you know, last week, you were talking a little bit about prompt pushback. And, you know, I think that's going to be something I really want to get into and talk a little bit about prompting when we get into our real, and we get into kind of our fireside chat, our conversation piece. But tell us what do you have for knowledge by today?
Teach me. Yeah, so for those of you who are linked in or perusing the internet on AI, you have definitely come across probably some ads for something called open evidence. As I want to talk a little bit about tools, not just that tool, but tools like that tool. And there's another one called Vera Health.
And these are, because we talk to spend a ton of time talking about LLM Scott, but there are specialty, rather medically specific LLMs that I think our audience should know about. I'm sure many of our audience already know. So, you know, we've been using chat, GPT and cloud. And, you know, I don't know about you, but every once in a while, something definitely feels off.
Like the answers are really confident, and it doesn't hold up. And you're like, you look at a reference at its sites, and like, you kind of double check it on PubMed. Is that really there? And, you know, I've been burned.
Like it, it's made, it's made up references. I have a pretty example. You know, we have a, we have a whole series coming out on AI implementation. And one of the preamble ones is something we call revenue per minute, and it costs per minute.
It talks about how do you really determine this? And when I was doing some research on this that come up with, and it said the average optometric, you know, revenue per patient was like 882. And I'm like, on what planet is that happening? Because I want to be in that practice.
And, you know, so it was one of those, I had to go back and double check. And, you know, that was a very example of reason. If I had just taken a face value, I'd been like, wow, I'm not doing so well. But nobody makes that per on it.
And this is definitely something that you see. It was like, hey, I slop. And that's what that would, it gives it a bad name. And we check someone's work.
I know this reference never existed, right? And so there's tools. Now, revenue per minute, other types of things, maybe, you know, not as serious. Still, I think a huge problem.
But when it comes to taking care of patients, we're talking about, or legal issues. We're talking about big issues there. And we, we can't afford to be wrong. So talking about two tools.
So one is open evidence. Now, this is, this is probably the biggest one. It's called itself like the clinical AI co-pilot for US doctors. And it's very different than like a chatbot.
So instead of being trained on the open internet, which is the way Claude or Jackie, Jackie, Jackie, PTR, it's, it's actually trained on medical literature. And so it's had these kind of impressive content licensing deals with journals, like, and the big ones, like, you know, the New England Journalist, medicine, JAMA, and all the specialty journals, the Cochrane database, and a bunch of other journals. And so if you ask a question, the answer is synthesized from primary literature with citations. And it's really popular.
I had a, my cousin is a physician. And he, he used open evidence every day. And they have, they have more than 25% of US physicians who are actually using it commonly. I use it every now and then.
I'm going to be able to, I'm not using it commonly. I use it every now and then. I'll sort of go my reasons a little bit later. But in December of last year, it's supported 18 million clinical consultation.
So it's pretty, it's getting, you know, almost mainstream. And I'm sure the, the numbers are going to definitely increase. And in fact, this earlier this weekend, the reason I want to talk about this is the American Academy Ophthalmology announced a partnership with open evidence. And it's rehabbing their preferred practice patterns and their clinical guidelines sort of be ingested by them.
And so that, that was just the first announcement. We'll see what, see what comes of it. The other two I definitely want to talk about is called Vera Health. And that's a newer entrant.
And as a slightly different philosophy. So I encourage, you know, we show a shop around, see what works best for us. This is different because it's retrieval first architecture. And so most of these AIS, since like, like open evidence, they, they get to citations and, and then they sort of generate the text.
And that's where you could theoretically have hallucinations. So Vera says it reverses that it searches a corpus of 60 million peer reviewed papers, guidelines. And then it does something interesting. It applies evidence grading logic, kind of like your Google search ring to the literature.
And then it writes the answer with inline site. And also has other features that other others, you know, like Jackson, he doesn't have has drug interactions automatically checked, dosing guidance, medical calculators, and has sort of CME built in. And so. And they also have some partnerships with the American College of Emergency Physicians, thinking that, because they're a newer company that I call, if we could get emergency medicine on board in that sort of fast paced, high high, high, high, high, high, high, high, high, high, high, high, high, high, high, high, high, high, high, high, high-intensity type of clinical encounters, well, we could then go to the other specialty so those are two general ones.
Again, the core advantage here is that the hallucination risk is materially lower. They're both grounded in actual literature. And then you have a corpus that's curated right? You go into the top journals, the new of material medicine, JAMA.
And so you have some confidence in what's being presented. They update their tools and then they have these kind of interesting workflow features that they're coming out with Like the calica it or CME, you know, maybe you could do it with like your own notes transcriptions that type of thing So I definitely encourage our audience to check these out um Of course, you know Follow their security policies. HIPAA they both state that they are Designed for clinical use, but something for for on it's definitely to check it out So I I am using these tools. I think they're really great I have not used them as much maybe as some of my colleagues interested in Scott Probably because I actually find clawed and gbt with the right prompting which I know we're gonna get into To be actually pretty good But I'm really excited to try these out and more in sort uh, you know, I just I just opened up count with Vera and been using open evidence But just to to use them a little bit more and excited to hear from our audience, you know, how they've used these various tools And what they've learned and both are free to sign up to encourage everyone to check it out at openemons.com and Vera Health And there's others doximity I think it's coming out with one and there's several other so I encourage us to all learn together here Scott, have you tried any of these like open evidence?
What's interesting because as I'm building my My uh software right now, you know, I am at that point where I'm doing treatment and we're looking at open um You know, I kind of open evidence is The can it give us what the true evidence-based medicine data is that's at least out there right and the problem with that I mean, I think it's great and like we always talk about around. It's another tool But I think sometimes we look at these like I look at some studies and go Yeah, that I know that's the result, but that doesn't make any sense in clinical practice Because then you break down the study and you know, you find the study wasn't that very good Yet other people are using the conclusion of the study to decide what they're gonna do So I still think it's a it's a tool right But at the end of the day, we're still gonna use our experience and sometimes our gut which is maybe the emotional experience To kind of go does that sound right? So I think the more information We have Which very health and open evidence will give us The challenge we still face with both of those technologies is it's based on a limited data set of what Resources have been published And there's a whole bunch of stuff that has never been published yeah Because there's no money behind it and so why somebody doing a study on it right and so I remember my time back at Baskham Where when I was doing my training and they just taught us how to just absolutely rip journals apart You know and find out what articles are really good and and I still think You know open evidence and I haven't really used very health. So I'm very interested in digging to that one is So are they given the conclusion or they really looking as the study valid, you know, and so I think we like we talked about with even the LOMs that we're probably getting to in a minute You know as I think we're still in the lifespan or life cycle of AI We're in just past the toddler stage, right?
I mean we're These are all tools that are gonna they're gonna evolve. They're gonna adopt. They're gonna get better We're gonna use them more and more and I for all the audience. I think you know jump in try this, you know and see What fits you, you know, I think we're right right and I about to get into our fires I would talk about different LOMs and I know that you know right you started out with chat GPT And then you've kind of moved a little more to Gemini and I kind of start out in Gemini and I've kind of moved a little more towards quad so Yeah, I mean, I think that this open evidence and very health we're probably gonna find that or maybe there's gonna be a next You know evolution of that or maybe they're gonna come together or who knows But I think the more data we can get to process and help patients understand and make better decisions for their own health It's all plus Yeah, so yeah totally agree For a prompting that you know, we were talking we've been talking a lot internally For the audience Rayhan and I have about like how do you prompt we we talk a lot and I talked to I got off the phone today with Eugene Shatsman who's been on our show for a couple different things And we got on to a spiel today about well, how are you prompting into what you want to do and you know, I said when I asked you, you know, how are you Prompting and I know some of the more experienced listeners they make all of this the way I do it when for some of the more inexperienced listeners I hope you go.
Oh, well, I hadn't really thought about that. So rant tell me how you When you sit down at LM Whichever you're gonna use How do you decide what to ask it and then how do you Oh Ask back or retort what they're saying or reference check or proof check or whatever Yeah, so first off there we we've talked about this many times and we have almost have published I think guidelines and you google or you can ask jgpd or claw now the guidelines There's so much out there and it's super sophisticated about setting all these sorts of things I sadly um, I I don't do all of that um, but I tried to give it as much context as possible in my first question I will and I have various projects that they know exactly what I'm looking for when I'm and I'm those like paragraphs But if I'm doing a new project, I basically will say I am whatever I'm an ophthalmologist and I'm seeing these patients and etc etc I try to give it a little bit of context and then I After I asked the question I I never stopped the conversation there. I always Push back or we talked about last time. So I will ask questions like Uh kind of even sometimes it's simple.
So are you sure about that like are you sure about this one factor? I've heard differently and I will Surprised by how often says oh, I'm glad you check let me search that and it searches and comes back with what I thought right And so I definitely push back a ton. I mean, I think that's probably that my my take home point for the audience is You never take the first thing it says it's trying to be ever trying to be your friend It's trying to be aligning with what you want think because it's it's trained Think of fan right they're all I think being very sick of fan. Of course, and so you have to almost push back and be almost a little rude to it to get to really push it a little bit How do you prompt?
Well, it's funny. You know, I was I've been thinking about this a lot as a and I think I told a couple weeks ago I've been building some agents and As I was going prompting how do I build this kind of agent? I started going man I'm spending a lot of time prompting and reprompting and I don't think my prompting styles very good And so I actually spent a couple hours going okay doing all that research You're talking like listening and all this stuff and go well, how do you actually prompt correctly? And I learned a lot and said maybe I haven't been doing it right And I kind of wrote down all that stuff and I've over the last couple weeks I've been kind of summarizing it because My marketing person Jess and some other folks are like okay, well, Scott, how are you prompting?
And I said I come up with some guidelines and so I've taken a kind of a lot of this and this is my opinion and I've Kind of fine tuned it and I'd love to hear from the audience that you have better ways to do it Then great. We're all going to get more efficient if you you know send emails to to me. It's scot at AI in iCare.ai Array on our e h a n at a i in iCare.ai We'd love to put your comments on the show and so we other people can learn from The way you're learning to prompt But I think first of all is I have to say okay, what's my task? What am I trying to do because I might be writing a paper I might be building an agent.
I might be the other day. I was looking for a recipe You know, I mean has nothing to do with my professional life I was trying to think what's a better way to build this system What's a better way to say this particular phraseology for the speech I'm giving And I think I had to pick a task and then I kind of started thinking rayon you know as I I was I got almost two Gemini centric I become too dependent upon a single source and granted it's learning me And I'm learning it and now when I ask a question it builds some of my other pre-existing prompts into it because it knows Hey, optometrist you're building this podcast series You're rebuilding this project and all of a sudden those things start showing up In my responses because it knows I'm kind of digging in that direction So I think for general system. I'm just general information. I'm still using Gemini a lot um, you know as I'm starting to do more of coding and development I'm moving more and more into Claude.
I can't say I'm fully embracing it yet, but I find myself probably 75 25 or 70 30 at this point Gemini versus Claude and Claude's increasing every day Um, but interesting enough I mean then when we're doing content for creation You know for some of the videos and stuff that we're creating for some of my other projects You know, I I use synthetic uh, I use I use a bunch of different stuff and I'm learning how do you prompt them is different than you prompt everything else Which ones um for which ones are you paying for that you like So much that you decided that you're you're You're getting as you're giving them. Oh, yeah, so I use I use the highest version of Gemini now Gemini pro and it's like I don't know. I think it's one. I can't remember off top of my head round But I think it's 149 a month and I know some people go.
I'm not gonna spend a hundred and fifty dollars for this But let me just tell you So worth it. I mean, that's worth it. Yeah, it saves me dozens of hours Roundover joking today and I know that you guys are gonna hear this for a couple weeks But this is May 6th today and we just got hit with a huge snowstorm up in the mountains in Colorado And so I've had the data sit and work on projects and I mean just today alone. It's probably saved me 30 or 40 hours of What would have taken me 30 or 40 hours?
I was able to do in four so is $150 worth that this is the best ROI ever right? ever But so I think that you ask what things and I I'm gonna pull up my browser because um and pull them down just and I'm not trying to promote any one project or any one software For any of the audience. I'm just telling you guys what I use um in terms of different projects. So like I said, I use Gemini.
I use Claude. I mean, I still use chat GPT on rare occasion But it's getting more and more rare In terms of how you do your pain for these three I'm paying for Gemini and I'm paying for Claude So yes, definitely paying for those two For a lot of our creation stuff we use design capital D Z I and E We use a lot of that for our video that we don't do in Gemini Vale or Gemini Nano banana So we do that a lot um for a lot of our presentations I use gamma for a lot of that kind of stuff and for those of you use gamma you can convert that into PowerPoint or into notes or into slides if you want um You know, I think I'm I'm kind of using I'm kind of becoming where I'm using more and more stuff depending on the tasks that I want But let me stick with Gemini for a minute rayon. I'm kind of go through that because you know I want to talk about how and you kind of mentioned it So now what I do is before I prompt I actually do what's called reverse prompting I say You know, I I need to develop this I need to do this. I need to create this task or I need to write this or I need to research this Ask me I ask the LM ask me Three four questions one at a time So you make sure you understand what I'm going to ask and I'm surprised that some of the in-depth questions It asked me about what I want like I was um developing this new educational series and I started going through and asking like Ask me the questions you want to know so you can develop the right scripting for these processes And it asks some really in-depth questions and like I said before it started referring to some of the other searches and chats I've done in the past going Will do you want it to look more like this educational project you do or more like this one?
I'm like well more like number two and I thought wow it remembers that it actually did this before and so I ask it one question in time and ask it to ask me three or four questions one question in time And I found out that that helps me build my next prompt My actual prompt much better because it kind of gets me and the LM thinking in the right Right direction like this is helps me develop a more specific Prompt and then I give it a role you know, and I found this is something I really started doing the last two weeks Rayon as I said act as a marketing strategy act as a COO act as You know a movie producer act as a script writer and when you give it a role I find that it changes the way the outcomes work dramatically in terms because now it's saying I'm going to look at it from this perspective and so um or look as a senior consultant right I've pumped in my spreadsheet from this last quarter That I keep and I said okay look through this act as a strategic consultant as Looking at revenue per minute or revenue per hour and tell me what I'm doing wrong And it spit out a true I mean more of the prompting I'm gonna get to in a minute It's spread out something that I probably would pay thousands of dollars for a consultant to spend dozens of hours to figure it out and it did it in five minutes You know that was pretty good So I said give it a role then give it context and I started asking this I want to build X which the reverse prompting I did in the beginning helps me really kind of define that because I want to do Why and why being the word why why do I want to do it? Well, I'm building it because I want to educate this group of people So then your status the next thing who's the audience right? So I've learned when I say why am I doing it? I naturally include the audience I'm doing it for because I'm doing it for Faculty at a university versus I'm doing it for students versus I'm doing it for you know high school kid presentation I'm getting ready to do it when the local high schools It's presenting in a very different format because I'm saying why Um, and then I'd say okay, this is what I needed to do and this is the three or four questions You know that you I've talked about this a lot right on is I get very granular About the exact task I want and the more granular I get the less Back and forth conversation I have to have less re-iterate or rest iterations I have to do And then I say what I want to look like.
Don't I want it to be a table? Don't want it to be a graph Do I want it to be a video presentation? Do I want it to be an image? And then I do it and you know, it's getting pretty good But then I think you know, I want to hear you because you talk about how then you keep asking it back and forth I don't do that very well.
I don't think and I'd love to learn from how are you asking it How are you re-iterating your questions to make it even more detailed because that's a part I don't think I do well Yeah, so this I think goes into a little bit, you know, I'm a A philosophy major so I love the Socratic method and so I do a lot of I don't know Maybe they're called like suppositional questions if I give it a claim. I'll say well I Suppose Suppose it were true if it says all you know this can't happen you have to create a net idea because X is the case I suppose X is not the case Um, and I just I just sort of push back I try to be kind of like A cynical kind of rude kind of a jerk person back You know, I'm not I don't think I say thank you that often. I just say well, let's let's just I even say well Let's pressure test this imagine imagine you are an investor on the other side of the table Imagine you were a cynical imagine and so because you want to hear The you want to hear the LM Tell you all the reasons why it was just what it just said why it could be wrong the previous On the previous prompt and then you use that to push back and then you sort of go back and forth the other thing I do is I don't know if you ever done this I a copy paste from one Chatbot to and I use a used a lot, but I've been so confused thought but I mean I'll go from gpt to Gemini to cloud and go back and forth And I literally say this is what jack gpt said this is what clause is and and and And yeah, and they and they converge or they where they push back from each other It's really interesting. That's where like some human judgment comes in you you kind of like have two students and you're the teacher Well actually Claude I think you're right here, you know, I'm gonna go with you Well, I kind of start out with the weaker stuff for like some of the coding stuff And I'm sure Google finds this out.
I'll probably get like blackball or blacklisted on this one But I don't think it does a good job of coding as Claude code does And so I'll start out in Gemini and then I bounce it into cloud and click goes no you wouldn't do it that way And I'm like really cool, you know, so I I agree. Thanks for bringing up as I do use them against each other More and more just to see what the other one's gonna say, but I mean, how is that any different than you know You and I have a conversation and you say something like, well, then I talked to one of my friends like brand told me this What do you think about that right all we're really doing is having a a waste smarter At least able to access the world Series of consultants to help us make better decisions, right whether it be what we're producing or however, so Yeah, 100% Yeah, and I think though I think the exact phrasing you just use there a series of consultants And that's the way I think is the healthiest way and the most appropriate way to be using our AI systems and we hope you know, we can consult with your projects as well to our audience You could find all our materials at AI and I care Dot com and of course, as Scott mentioned, please feel free to reach out. Please reach out to us Scott SOT 1T at an I care dot AI and rick on our EHA and at AI and I care dot AI you can find all our podcasts our video innovator series You know, we try to do really interesting. I just did one with John them John Lee over at we used to be at UCSF on myopia A great talk there and I know Scott you've done some really fantastic talks recently So please check us out and look forward to chatting you chatting with you next time