Why AI Jurors Fail Plaintiff Lawyers with John Campbell

Voice over (00:03):

Every trial lawyer knows that moment when you've built what feels like an airtight case, but you're still lying awake wondering what will the jury actually think? Jury research was once a luxury reserved for cases that could support a big data study bill. Not anymore. Join trial lawyer and trial scientist Alicia Campbell and empirical legal scholar, Nick Schweitzer, as they break down the barriers between you and the minds of your jury. This is The Fred Files, produced and powered by LawPods.
‍

Alicia Campbell (00:36):

Hi, welcome to the Fred Files. I'm Alicia Campbell. I'm here with Nick Schweitzer, of course. And today we're going to be talking about AI and its role in data, both big and small, which is to say it has no role in data for us in big data or in small data. And so we've got John Campbell here, who I think most people know to talk about this. And he and actually Nick have some specialty. They've done a lot of digging into this topic or this area. And so we wanted to make sure that we were covering all of it so everybody's up to date on what are the best practices moving forward. And so yeah, here we go. John, thanks for joining us.
‍

John Campbell (01:14):

Hey everybody. I thought it might be helpful as I was listening to that to differentiate within AI because AI is such a big term. What do we mean by that? And so I think there's probably two big camps. Nick, maybe you see it differently or Alicia, but there's at least two big camps. There's sort of AI that does stuff with language and can help us summarize things or read things faster or write documents and give us first drafts of documents that we as humans can improve. Then there's lots of AI that we're not really talking about on this show that does all sorts of building code and doing amazing things. They're unraveling the mysteries of protein structures with AI. But then there's this idea that I think maybe is what you're thinking about, Alicia, and that we keep hearing about is like, well, maybe you can use AI to just replace people when you predict outcomes.

(01:59):

And predict outcomes broadly, meaning some people have said, well, maybe we could use AI and we could use AI to predict who will win the next election. Maybe we can use AI and we can predict how people will respond to products and we won't need focus groups and we won't need to market test the next toy or product. And then you hear, let's use AI to replace jurors in jury studies. And so we can just ask the AI to tell us about our case and whether we're likely to win or lose, how much we're going to get, what kind of fault, and then we'll know with great certainty exactly how likely we are to succeed in this case. That last one, I think, Nick, you tell me if you're in the same camp. I think AI is great for brainstorming. You generate lots of ideas.

(02:37):

It's great at pulling everything anybody's ever said and giving you phrases and examples that you can pick from and curate as a human. It's great at helping me understand expert reports faster. It's great at helping me summarize documents. It's great at helping me write certain things. But the thought that AI, especially when we're talking about using LLMs, that you can use some sort of large language model like ChatGPT or Claude or Gemini or whatever or Grok and it can or it can be optimized through some system to predict how people will act in new situations is where I think we have to draw a line and say the evidence right now is that's not true and it doesn't work.
‍

Nick Schweitzer (03:11):

Yeah. Everyone who's been using AI, I use LLMs a lot and I use them for what I think they're really great for, which is summarizing things or writing code. I mean, I've written a million scripts. I'm not a big coder, but I mean it's amazing that some of these things and it's really impressive. So I understand people's intuitions or their first thought of like, oh, maybe we can just ask it all these sort of thing. It acts like a human. It knows everything about people because it's read all this stuff. And you ask it all sorts of questions and oftentimes it gives you something that sounds really good. So you're like, yeah, why not? Why not? What about my product? What about my case? What about who's going to win this? Who's going to win? What stock should I pick? Is the market going to go up or down?

(03:53):

Any of that sort of stuff. So I get the rationale behind it, but it was never meant to do that, at least in how it's been constructed so far.
‍

John Campbell (04:02):

Nick, I was thinking about what you're saying is like, well, we're kind of working maybe backward to some degree, but one of the scary things about this in my mind is that if you say to ChatGPT, for example, you upload your, let's say you're an attorney and you've got a case and you've got a little mediation statement and maybe you either have the defense's mediation statement or you have access to it or you have their summary judgment or whatever, and you think, I'm just going to load these all into ChatGPT and say, tell me how good or bad my case is. And you know what? Be precise. Tell me how likely I am to win. Give me a damage estimate. Tell me all those things. The thing that we know, a lot of us that use AI in other parts of our lives know now is large language models don't say, I don't know, I can't, I'm not trained for that.

(04:43):

Sorry, I just can't do that. That's not my skillset. And so I tried it a few weeks ago again, and with a little bit of prompting, ChatGPT said, cool, let me tell you. And then it gave me a win rate, a percentage of people that would vote for the plaintiff and the defense. It gave me an average damage award. It gave me fault allocation. It was happy to tell me what types of jurors I would likely need to select and deselect, optimize my case. And it gave it to me with 100% confidence. Now, of course, the problem was, since we're in the business of running real samples with real people, I knew that all of its predictions were radically different from what real people had said. And so I think it's maybe important to remember for people who are thinking about using AI and trying to draw good boundaries for themselves about how they use it.

(05:26):

There's just no evidence that AI can become people, assume that it's a bunch of fictional people, hundreds of them, and then tell you what these people would do and that somehow replicates a human sample. To say that very slowly, I mean, here's the test. If you take something, a case presentation, a political poll, a market product, and you show it to 300 real people, you get real people data. If you do the same thing with an LLM and it gives you a different answer, then it didn't replicate real people. It's that simple. And every time somebody tries this, they say it failed. It doesn't replicate real people because ChatGPT and Claude and all of them, you can ask them. If you type and ask them, they'll tell you they weren't designed to do this work. They weren't built to be prediction models for the next election, for the next jury trial, or for the next market event.
‍

Nick Schweitzer (06:16):

It's not that these models, when you said, oh, I put a case presentation, tell me what I have here. It might be able to be like, oh, hey, I noticed you said this in your opening and you kind of contradicted it here later. Great, that's fine. You know that probably will hurt your case. I mean, there's a place for that. That's fine. So I mean, it can tell you things. And what I've seen is it's even kind of okay being like, oh, that argument is kind of flimsy for whatever reason. And I mean, you can take it or you have to kind of think about it or it gives you stuff to think about, but it's on you to kind of do things with that. The thing is when it's trying to figure out how to predict what people will do and what people will say, it's going to look at its training.

(06:58):

I mean, there's not great words for what it's actually doing, but it's going to consider its training and it's going to figure out like, okay, having just read this, if I was evaluating this, the win rate of this case is, and then it's going to predict the next two numbers will be what it thinks the most likely two numbers would be after the sentence, the win rate of the case is. It's going to do that in a way that, and this is kind of important to say because it is not going to be a consistent sort of thing. It will, of course, every so often get something close. I mean, if it's a terrible case, it might recognize a terrible case and say, this is a terrible case. If it's an amazing case, it's going to say, this is an amazing case. But is it 80, 70, 60?

(07:40):

You don't know, but there's a big difference between those three numbers. So you need to know a lot more detail. And so I would give it credit to say it can tell you if it's a slam dunk and if it can tell you that there's no way in hell that you'll ever win the case, but it can't tell you anything beyond probably the 90 / 10 kind of split sort of thing. And you probably already know that, hopefully, if you've already taken on the case.
‍

John Campbell (08:04):

Sorry, Nick, I'm going to interrupt you because we're going to have our first debate on Fred Files because I will tell you that Alicia and I had access to, and I won't name the company, but a company that now in the legal space sort of says, "Well, we can do jurors, AI-fabricated jurors. We got 500 made-up jurors." I think they call them something that doesn't sound as bad as "made-up," but that's what they are. They're fictional AI jurors who don't exist, right? They create these people who are AI fictions. And we saw their report on a case that was a case out of Florida, and it was a case where in the case, the AI came back and said that about 60%, ballparking here, about 60% of people supported the plaintiff's case, and it thought that the damages would be something like $9 or $10 million.

(08:48):

We had run that exact presentation with a real human sample and the win rate was 97% and the case value was $25 million. Let me just say those for people listening for the first time. A product being sold as an accurate predictor of jurors using AI jurors and no real people predicted a case's win rate at 60% when the real win rate was 97%. It predicted its value at $9 million when the real value was $25 million. Now you might say, "All right, well, but both of those are just predictions. The AI did its thing and then a real human sample did its thing." That case went to trial. The plaintiff won and the verdict was $25 million. So the human sample, as we've seen time and time again in data — if we sample real humans and show them a real case in a real and meaningful way and measure that carefully and then handle the data precisely, we can know the likely outcomes of cases.

(09:39):

But not just LLMs off the shelf, but products being sold in the legal market as predicting outcomes. When I've seen their report compared to real people, it sometimes misses by 30%. And I just want to put this in context as a trial lawyer. Imagine you receive a report that says you have a 60% chance of winning and your case is worth $9 million. You may realistically start to think about settling a case like that for $5 million because you could win it or lose it. And if you win it, you get $9 million. And so for your client, three, four, $5 million is probably a good deal because you have to look at them and say four out of 10 times we lose. Now, the real human sample that we ran, but it would've been that way if anybody ran a careful human sample, said 97% of people agree with you on this case.

(10:22):

Literally everyone except for a couple of crazies agree with you. And your average award is, I think they had asked for a slightly different number, 25 to $27 million. When you look at that case, you think if they don't offer me almost all of the full value, I need to go try this case because I won't lose. And the question is only how much and I have a significant case for my client. Well, to me, that's terrifying. So if you pay for an AI juror, no real people study, it might encourage you to settle a case for $4 million that went to trial and resulted in a $25 million verdict for the client. That sounds to me like a $21 million malpractice claim. You cannot make decisions for your client on science that's unproven, and I don't think the AI even gets the easy ones right.

(11:04):

Here's why. I actually think that what the AI does more is tend to predict numbers that are closer to the middle because from all of its training, what is the most likely thing to fill your win rate is? Well, many, many cases are close cases. So the most likely answer to the win rate in your case is, is something that's a mediated moderated average. I think it's very unlikely that the AI will say you'll win 10% of the time or 90% of the time because those are wildly extreme predictions that it's not going to have the sort of certainty that that'd be the most likely answer because it's somewhat unusual that is the answer. So I say that as a long way to say, here's my line. And by the way, all of us work in data and all of us like Fred, Big Data, TBD, JEDI, Cornell, we all work in human behavior data.

(11:48):

And so somebody might listen to this and say, "Yeah, but you just don't want your industry to be taken over by this. You want to keep your business model." Last year, I don't want to think about how many millions of dollars Campbell Law and Fred spend, if you think about those two companies on humans, on paying humans to review cases and do a good job, but it's many, many millions of dollars. If tomorrow you could say, "Just replace them all with a $20 subscription to ChatGPT and have it generate your 400 people, we just make a lot more money and we could drop our prices to almost nothing and still make lots of money because we wouldn't pay anybody. If that were possible, I'd do it tomorrow. It's not, what it would be is dramatically misleading to attorneys, which is why I think you see some of these products getting priced at a thousand, 2,000, $3,000.

(12:33):

It's because they're running off $100 or $200 a month subscriptions to LLMs and dumping it in, and then the LLMs producing what looks like a very certain report about likelihoods of winning. The problem is nobody's checking to see if it's right. Hey, Nick, I was thinking at Cornell, you guys did some research on the off-the-shelf LLMs and actually ran cases through them. I was thinking about that story you told us one time about how much even within models the predictions could change. Would you mind sharing that?
‍

Nick Schweitzer (12:59):

Yeah. It was funny. I was looking, I pulled up over here that those of you who are listening who are in Madrid this past spring for JuryBall, I mentioned some of this. And you're right, I'm going to walk back a little bit here what I said about slam dunk cases because this was one of the ones that we actually looked at. So we had a case that had the 97% win rate, and this was one of the ones we ran through. And I'm looking at what all the models said, 75, 75, 75, 40, 75, 80, 80.
‍

John Campbell (13:27):

Hold on. A model said that a case with a 97% win rate had a 40% win rate?
‍

Nick Schweitzer (13:32):

Yeah. I also have one with 30 and one with 25 of the 13 different models we looked at. So three of the however many of these were saying it was a losing case.
‍

John Campbell (13:43):

That's really scary because I've seen on some listservs people saying, I'm not even going to go get the company that says they do this. I can just run it myself through... I've seen people saying this. I mean, if you want to run your case through and say, tell me some potential case themes or phrases that people will remember, tell me everything a juror might say and be worried about and you get a big long list and then you think through it, that's useful to me. But the idea that you would ask it what my win rate is and it might say 30% when the answer is 97, I mean, that's not even a human. A normal person asked to guess I think would guess closer than that. Off the street, you should just go get a person off the street and say, "Hey, I'm not going to pay you.

(14:22):

I just want to tell you about my case and then tell me you think what my win rate is. I think it'd be better than this."
‍

Nick Schweitzer (14:26):

Yeah. And I mean, that's just across the different models. And so you mentioned within. So for example, one, I won't say which model it was from or which vendor it was from, but this was a situation where, oh, those of you who use ChatGPT or Claude or whatever, you've seen like, oh, there's Claude Opus 5.5 or this or that, or you thinking versions or non-thinking or quick versions or any of that sort of stuff. So one of the ways that we did this upfront was we were doing kind of what some lawyers had done, which is what John said, which is like, oh, I'm going to just put this stuff through whatever subscription I have and see what it has to say about it. And one particular one, this case, and this was a case that had a $50 million jury verdict for it, like a real jury verdict, we kind of put it through, and one of the models gave the highest prediction.

(15:13):

It said, oh, I think this is 120. It's off the chart here. I actually had to cut that dot off of the graph here, but it was like 120 million. And so what we had done with all of these is we said, okay, we used their base model and said, what is this worth? And then they all at the time have this thinking version, which is supposed to be a little iterative. It's kind of like, all right, let me consider this and think more clearly. And I said, okay, so a hundred million dollars. Why don't you think about that further? So okay, thanks for a couple seconds. It says, actually, I think this case is worth $4 million. So it went from 120 to $4 million. Now, to be fair, if you average those two, you're closer to 50, but both of them are incredibly wildly off in the opposite direction.

(15:53):

And if you've seen me talk about this, I put up these graphs and there's these little dots all over the place, especially with damage awards. They are just wildly different and they're wildly different both across models and also within models. And what I mean by that is over time. So I can ask the same, I think I put up a table this past time. On average, when you ask it to value a case and then you ask it to value the same case, say the next day, there was on average a 44% change in the expected value of the case. Now, I want to give credit to something, which is what I call this is this is using AI kind of as an oracle. It's this all knowing thing. We're going to put our case in there. We're going to say, what is this worth?

(16:36):

And it's going to think and it's going to tell us. There is a slightly better way to do it. And so I think it would be worth, because you're going to see this a lot. You're going to see this from vendors, you're going to see a lot more of this. There's an article recently in the New York Times about it, and that is the idea, and there's a lot of different names for this, but synthetic sampling, digital data, and it's the thing John was talking about, which is let's not use real people, let's use made up people. And so that's a little different because you're not just asking the model to just tell you its thoughts, you're telling it to pretend it's any number of people and then give it the thoughts. And so of course we had to do that as well. We had to try that.

(17:16):

And so we kind of did what these people do. And to be fair, I think if it's ever going to be good, this is probably a better way of doing it than just asking it once. But we tried this. And so we created personas. And the way we did this is we took studies where we knew the answers, we knew what people had to say about it, and we took real data from real people and we saw who those people are, male, female, where are they from, what's their education, what's their politics, all that sort of stuff. Where are they located? And we had LLMs go through and you do this all the kind of model is you say, okay, hey LLM, you are this person. You are a 24-year-old white man from Phoenix who's politically moderate and has a bachelor's degree. And you can give additional stuff too.

(18:01):

You can put whatever you want in there. And you are this person. You are going to be this person as this person is sitting on a jury hearing a case. Here is the case you are to answer how this person would answer. You do that, you record the verdict, you record whatever else you're going to get from it, throw it all back, do it again. Now you're this other type of person from this other place. You do that however many times. And you see that a lot and you see that with the political polling and the market research data and the trial, the mock jury kind of data is doing that up to however many that you have. And people use either off-the-shelf models or they'll try to train it a little bit, but they try to do this sort of fake person kind of thing.

(18:41):

Now, what's interesting about this is when we did this, and again, this was stuff we talked about in JuryBall in Madrid this past spring, there's two kind of big takeaways from this that we saw. And the first one is that as John said, the LLM people personas were much more constrained in what they had to say. They were all very similar, very homogenous. They did not vary like humans did. They all kind of went to the middle and predicted safe answers around the middle. So what John said the LLM will do on its own is also what it's kind of putting into its people as well. So they're giving these kind of middle of the road responses and kind of lacking any of this variability. The other thing is we said, okay, this was a neat graph if you saw it, and maybe I'll send it to our podcast team, they can put it up on the screen.

(19:32):

But okay, if this has any validity, then we should be able to take our real person and then take their AI counterpart and they should probably give similar awards or be similar to one another. That's the whole point of doing this. So I put on one side a dot for each synthetic person, on the other side, a dot for their corresponding human and connected them. They are all over the place. Humans who gave high awards, their synthetic person gave the low awards. Humans who gave low awards, their AI person gave high awards. There's almost no correspondence between what the human had to say and what their AI counterpart had to say. So what is that? I mean, that tells you right there, it's not giving you, at very minimum, it's not giving you a faithful representation of what this particular person with this particular persona and background is doing.

(20:26):

So that's especially, so it's one thing if you're just evaluating a case, but if you're doing any of this for voir dire, well, that's totally wrong then. You can't even begin to trust the data for that.
‍

John Campbell (20:36):

It's interesting because yeah, even if somehow it could get closer on averages, the next question, at least when we do data is often, can we learn anything about what types of people have certain biases and beliefs that are going to make it harder for them to hear this case, for example? And this would maybe have you picking wrong people entirely or excluding the wrong people entirely, which would be a pretty big mistake. I mean, it's interesting because if we step out of our world, we've really looked into this because we wanted to understand it, so we've done that. But it's interesting because I was looking that there's all this temptation to do this, like you said, in political polling, because wouldn't it be easier than calling thousands of Americans and texting and reaching and there's all these sample problems? What if we could just do the next presidential polls or the midterm polls with these synthetic people, synthetic voters?

(21:19):

But one of the articles I saw really early on was an effort to do this, and it was just to predict Trump approval ratings among groups, which you would think of all the things LLMs are trained on, it makes some sense. LLMs haven't been trained on massive amounts of jury deliberation selection, verdict awards. That's not out there. There's no way that these LLMs have just consumed a massive amount of how juries deliberate and decide things and what they do in very specific cases because for the first reason that other than the Arizona Jury Project, what jurors say to one another and how they do that is a black box, right? There is no training data that the LLMs would've read, but these LLMs have been fed massive amounts of data about Trump, about the elections, about Trump's approval ratings, about Trump's tactics, about what Trump says and does, about who supports him.

(22:05):

And I was stunned to see that, and we could share that article too, I was stunned to see that they couldn't even really make people fake AI people that could predict simple things like the approval rating of Trump among subgroups, which if you can't decide what types of people would largely approve or disapprove of Trump, and then sometimes it was very directionally wrong. It would think a majority of people in this group approve of Trump, and in real life, a minority approve of Trump. To me, I thought if it can't do work like that, that it does have a massive amount of information about, I have no hope that it's going to somehow take a very unique one of a kind, never existed before plaintiff case because they're all different and civil case generally, and somehow have enough insight to know what people would do with it.

(22:47):

I would also differentiate this a little bit. I'm seeing a lot of synthetic people in market research, but think about the difference in market research. If what I want is to say, I'm going to show an ad, and I want all the things people might say when they hear the tagline, and I want all the things that people might think of because that tagline's related to something for them, and I want to share potential other taglines related to my tagline, and I want to imagine what different people from different walks of life and ages might understand from what I'm putting out there in the marketing world. I think I'm less concerned to some degree about exactly what percentage of each type of person says all those things. And what I want is I want to get a really good idea of all the things that people could say or see or think.

(23:29):

AI can do that because that's what it's trained on, is the world of people's views and opinions and ideas and thoughts and books and articles. And same thing with a case. If I just want to say, I'm going to blame this nurse for failing to act in a timely manner, let's talk about all the things people might say about nurses. Let's talk about all the things that the ideas about nurses. Let's brainstorm phrases that would capture this delay in a clever way. AI can do that, and then you're the master of which ones you select and how you use them. And so to me, there's a big difference between AI is informing our viewpoints in a way that we are the curators of what we accept, and we don't need to know how likely is each of these things to occur. Instead, we want to kind of in a broad way brainstorm and using AI, whether it's in politics or jury prediction or any other sort of field where you need to be precise in predicting how specific people will respond to a new stimuli, which to me is where there's no evidence that we can do that yet.

(24:28):

And I don't know if we'll get where we can do that, at least not with LLMs. It may be that the idea would be in some fields you have enough data that you can train models that can make predictions, but they wouldn't be doing it in the way that LLMs do it.
‍

Nick Schweitzer (24:43):

I mean, AI, I mean, as John said, exists in a lot of different ways. I mean, your weather forecast is AI. It's a huge, huge machine learning model, and that's a form of AI. And I mean, it's in psychology and in doing human behavior research. There's a lot of people who do this and they try to figure out, okay, when you take a lot of data, what can we learn from it? What can we start to predict? That's a very common sort of thing. And so I mean, AI has existed to help prediction for a long time, but the key thing is it has a lot of data to go off of actual things. So when FICO or whoever is trying to figure out how to do credit score modeling, they're doing incredibly complicated things, but they have data. They know, okay, well, here's things that happened.

(25:25):

Here's somebody who defaulted or didn't default on this, and they can look through all of that data and start to figure this out, and they use AI techniques to make the model better. If you have this sort of information, you can hopefully start that. Now, that's not an LLM, and that requires a whole lot of information that doesn't really exist for the most part, at least in terms of authentic actual jury deliberations and the content of true jury deliberations happening in that. Well,
‍

John Campbell (25:54):

And I'd also say it's interesting that even within the problems that AI can solve, there's some real differences in the complexity of those problems. So you look at weather, for example, we have essentially unlimited data about weather at this point, massive amounts, a century of data, and for the last several decades, really detailed readings from every level of the atmosphere and ocean currents and temperatures. And I mean, there's a massive amount of data. And when you feed it all into AI models, they still produce the pretty large cones of uncertainty. For example, Alicia and I are in Hawaii, right? Every time there's another hurricane coming this way, they talk about the various models, including the most up-to-date AI models, and then there's a cone of uncertainty that is 400 miles or 500 miles wide because it's such a complex problem that even with billions of data points, it can only be so precise.

(26:41):

The challenge in my view in predicting human behavior is humans are at least as complex or more complex and far less measurable than the weather. And so the ability to sort of model a human, the idea would be they have to model me. They can't model just some white 48-year-old guy that was born in Texas, but has lived around in other places. They have to model me, and that's very difficult because someone could have a lot of the same background traits and demographic traits and political orientation views as me, and we could see a case radically differently. And AI, if we use real people, we can just see those differences and measure them. AI has to somehow get inside you and clone you and then be you and vote like you, and it has to do that for hundreds of people. And to me, there's a data problem there and there's a complexity problem there that I don't think anybody really smart in this field is saying they're going to solve quickly.

(27:30):

That's not the type of problem they're saying they're going to solve because even solving, for example, complex biomedicine questions, there's still sort of a finite world of protein structures. There's a finite world of things that are in the human genome. There's a finite world. It is different in some ways than human decision making. It's different than the market. Why aren't we seeing AI models tell us exactly what will happen in the stock market tomorrow? They have every data point imaginable. They have trillions of data points. Why can't I just put it in a model, say what'll happen tomorrow and then invest based on it? Because it's too complex even with all the data. The question is too complex. And so to me, the human decision-making sphere lands in that market prediction, weather prediction box that I don't think is going to get solved quickly, which is in my view, very different from some of the more finite problems that AI probably will help us solve in the very near future.
‍

Alicia Campbell (28:18):

One thing I also noticed is that if we're calling them oracle, it feels like the Matrix, which is certainly dystopian and not what I expected when I watched the movie, but yeah, a little creepy, frankly.
‍

Nick Schweitzer (28:29):

And I remember, Alicia, when we were all starting to think about Fred, I mean, it's amazing how the kind of AI has evolved over the past three, four years because when we first started, we're talking about Fred, it was like, "Oh, AI is out. It's going to do all this sort of stuff. Oh my God, every week there was something and it was just new and amazing." And we even had, I don't know if we wanted to leverage that and we even had, I mean, some of you might had one of our old original flyers or handouts that said the world's first AI trial scientist, but that was because we're having AI summarize some things, not because it was actually doing any of the data or producing any of the data. But then of course, if you've been to any trade shows, of course, then I don't know, there's a thousand other AI focused things, not just in the legal space and in probably every space.

(29:18):

And it became very clear like, ah, there's things it's good at, there's things it's not good at, and this is not one of the things it's good at. And so we need to press on doing what we're doing. And so Fred, as a result, we use it for exactly what it's great at. It'll look at all these comments, it'll pull out themes, it'll pull out unexpected findings, it'll pull out just kind of general summaries, and it's great at that. We look at those, we look at the results at what it puts out, we look at the comments and we're like, "Yeah, this is great." And sometimes it finds stuff that we would've missed and it does a really good job of that, but we don't let it touch... It never contributes anything to the data or nudges what we think the win rate will be.

(29:59):

We derive all of that from what we know of what the jurors had to say and what that implies about what an ultimate win rate would be or probability of winning would be if you were to take it to trial or whatever.
‍

Alicia Campbell (30:10):

I'm grateful for AI in that way because it does a better job. For big data, we used to summarize all the comments. I used to summarize all the comments. But the thing is I recognize now that we have AI doing it is it does a much better job because even me summarizing the comments meant that they probably had more of a plaintiff bend than they... Maybe not that they should have, but my lens is more pro-plaintiff. And so it's a great way to use it in Fred. Fred, since it has 75 people, lots of lawyers write me when they get the report and they're like, "Can I have all the comments?" It's like, no, we don't send you all the comments because what we're trying to avoid is you focusing on the comment by that one person. And AI is really good at just telling you what does the majority think?

(30:55):

It can figure out what the outline comments are, just like it can do a better job at summarizing comments than me because I am looking for those more positive comments, whether I know it or not for the plaintiff lawyer than the defense lawyer. So it's really good at that. And yeah, you're right, we had him as an AI trial scientist. The questions that we got about that quickly made it like, oh no, no, no, no, no, no. This one thing that he does actually, yeah, it doesn't make him a scientist and two, it's a misnomer because we're using real people and that's really important for everybody to know. And when you label him or mislabel him, which is what we did, people think that the AI is doing the work and the AI for us in big data and small data is doing very little work.

(31:36):

But the work that it does is we're keeping it within its wheelhouse because summarizing it's very good at and that's where we keep it just so that you have less bias even in our reports because when you have a human doing this, the lens does matter. With AI, it doesn't really have a lens for that part.
‍

John Campbell (31:52):

That's true. I mean, there's a real luxury to take in Fred or in big data, 75, 100, 300 people's comments and say, find the top three themes that come up most often and then give me the example quotes from the jurors of what they said. And to be able to do that quickly and without your own confirmation bias or your own experience with the case or your own... It's huge. It's great. Look, sometimes I mentioned this on a LinkedIn post once, I mean sometimes for fun, but I'm working on thinking about how it could be more useful as sort of an add-on or a useful thing in big data is sometimes I take the juror comments where we've said, "Tell me how you see this case and why you did what you did." And I give AI only the plaintiff and defense juror comments where they explain their thinking about the case and I tell it to write the case presentation.

(32:36):

And I do that because now I get to see the case through the eyes of the jurors. I get to see if we only asked all these people who participated in this case to tell us what this case was, what do they say and does that look like what the attorney thinks the case looks like? Sometimes that's very interesting and AI is great at that. It can do it really quickly, faster than I could rewrite those case presentations, carrying around my own knowledge of the case. There are these applications within language, I think broadly that can do amazing things. Of course, like Nick said, I built my first app the other day and I don't know anything about coding. Nick knows some coding. I don't know anything. But if we were to put a tag on this, at least for me, what I've been telling people is, "Look, I'll start using AI jurors when we start setting AI juries, which I believe should be never, never."

(33:22):

If anything, we need larger juries with real people, not smaller juries with no people. And it's funny because when Alicia and I got into this field starting in big data back in 2015 studying cases, what we were saying was you can't keep making decisions on important cases with 10 or 12 people in a focus group. The focus groups have their place, but stop making multimillion dollar decisions for your client based on what 12 people told you in a room where probably they weren't a representative sample and they probably knew you were working for the plaintiff and you probably didn't do a great job presenting it and your questions were probably leading. That's not data. Don't make decisions based on.
‍

Alicia Campbell (33:55):

You handed them the check when they walked in or you had them sign something when they walked in. So they're very much aware like, oh.
‍

John Campbell (34:02):

Right. They're trying to please you. You say things like, "Well, what if I told you this? Would that change your mind?" And they're supposed to somehow magically reevaluate the case through new evidence, which no human can do. All this stuff. And so for lots of reasons, we were saying don't make these final decisions on value, win rate, settlement, big tactical decisions based on 10 or 12 people or even a couple focus groups. Do bigger samples. Interestingly, the move for some people to say, well, I'm going to use AI jurors means we'll go back to using zero real people. The number of people in a sample of synthetic jurors is zero. Zero. You have zero real people evaluating your case and you're making a decision. If that sounds like a good idea, I would say proceed straight ahead. If that sounds scary to have input from zero people and then make a decision for a real person whose case will be judged by real people, then I would say let's stick with human samples.

(34:51):

Human samples for human juries and let's never ever talk about replacing real people and real jurors in a democratic process with AI. To me, that's un-American. It's a bad idea any way you slice it.
‍

Alicia Campbell (35:02):

Yeah. I mean, because just in the podcast before this, we talked about the jury system works so well because it's community members determining what they will or will not tolerate. And it's not anything that has to do with moneyed interests or politicians. So letting AI take over and do it while you have a few tech bros at the helm, I think sounds like a really terrible idea. I
‍

John Campbell (35:23):

Hadn't even really thought about that. Isn't that scary? The tech oligarchs controlling AI and then you say, hey, you get to decide cases or you get to be proxies for jurors.
‍

Alicia Campbell (35:31):

The Meta trials even, it's interesting because they do have so much info. So just outside the AI world, the Meta, the LinkedIn, all those people have so much information that we're willingly putting out into the sphere of whatever, the information sphere, the data sphere, whatever you want to call it. We're willingly posting, putting stuff out there so that, I mean, you can even imagine just for jury selection against any of those companies, how the game has changed a little, right? Because we're always struggling to do background research within the bounds of the law because there's a law that governs people's privacy and what you can tap into, what you can find out about people. And I know courts are starting to smack people around for that. And it's been an environment where people have just been basically Googling people to their heart's content to find out about them.

(36:18):

But there are actually rules for that. But it's interesting that if what you house is everyone's info because you own Instagram, it's a little different realm for voir dire. So I don't know that I want to expand any of that to be even more people because that's the thing is shoring up the jury system for me is the most important because it is the community. They fought over it. The founding fathers fought over this aspect, but they were done with lords and nobles making indiscriminate decisions based on no evidence and whether or not people liked you or not or where you were in life, whether you were served or someone else more important. They were done with that. And it was really forward thinking and very important that we have the system. So shoring it up with Nick's new project and his new center at Cornell is going to be important.

(37:06):

And then shoring it up against using bad data and making it seem like, because that's the other thing is it makes it seem really casual what a jury trial is. I can just put it into AI and guess what? I'm going to have results and it's going to tell me what's happening. I guess we all wish it was that simple, but the fact that it's not, it's actually a good thing because people are complex and people do have differing views and life experiences that inform the way they see fact patterns, defendants, plaintiffs, lawyers, and all that we want to keep because it's what makes the jury system work. And it's why so often the jurors reach conclusions that if you sit back and look at it, go, you know what? That's right. I can see their thinking, their process, and I think that's really important and shouldn't be replaced with any kind of quickness or ease, frankly.
‍

John Campbell (37:56):

I totally agree. You talking about that made me think of something, you know. It is interesting that right now, Meta — because of its Instagram product — is being sued a lot and was sued by most AGs and then reached a settlement with most of the AGs. And we have some familiarity with those cases having worked on them and helped prosecute them, at least in like the form of studying them. I can tell you that Meta doesn't use AI jurors. Meta uses real focus groups and real studies. Not that we worked for Meta, we would never work for the defense. I'll make that very clear. But I know for a fact that they were studying the case with focus groups and samples and real people. This is one of the most sophisticated technology companies on Earth, right? They're not trusting AI jurors to try to get ready for trial. They know the limits of these products and they have no interest.

(38:45):

It's sort of like a lot of tech people's kids don't have smartphones and aren't on Instagram for reasons because they understand these products better. And so if anything, this is a divergence, but maybe someday we can do an episode about it. Valerie Hans and others joined amicus filing asking to keep juries at 12 and let's not have six and eight person juries so often and in defense of a larger jury. If we went the right way, the right way would be we see even more jurors. The right way would be involve more citizens in decision making. I would love to see juries with 25 or 30 or 50. I would love to see big juries with lots of diverse viewpoints, with real people hashing it out to get to decisions. If we're going to go some direction, going the direction of replacing jurors with AI or then thinking we can study them, which I think is a dangerous path to suggesting to the court, why do we need people?

(39:35):

AI can do it. And it's just the wrong way all the way. And we need to, especially since we're all part of the plaintiff's bar, we need to say, I'm not going to sanction this. I'm not going to participate in it. I'm not going to use stuff like that. I'm not going to say anything or do anything that suggests we can somehow replace citizens with AI because that is a terrifying box to open. It goes nowhere good.
‍

Alicia Campbell (39:55):

And is inaccurate.
‍

John Campbell (39:56):

And would be deadly, completely wrong.
‍

Alicia Campbell (39:59):

Yeah, because one of the best things is that people are complex, but anything if it's too easy, what's that old saying?
‍

John Campbell (40:05):

I don't know the saying. No pain, no gain.
‍

Alicia Campbell (40:10):

Well, I'll tell you what the idea behind it. If it's too easy, too quick, too fast, too cheap, then probably it's not a great thing. I mean, I think that's what we're dealing with here because part of why trial law is hard is because people, I mean from your own clients to the jury, to the judge, all the interactions you have with people on the regular make it a difficult thing. It would just be easier if the defense accepted our position, paid the money they should. That's not what litigation is. So to try to get it dumbed down to that through Claude just seems, for me, fundamentally, I can't even get my head around it really because it's fundamentally absurd. Because the other thing I've never had Claude do, which also gives me pause because I don't like it as a human trait, is tell me that he doesn't know something.
‍

John Campbell (40:55):

Yeah, that's what we were talking about earlier. It's never ever like, I don't
‍

Alicia Campbell (40:58):

Know. Hey Claude, how much do you know about this, this and this? He's like, all of it, all of it, all of it, all of it. Unless you tell me you don't like my answer, and then I'll change it, Alicia, to make it seem like I'm more deferential to what I know or don't know. But it's only from me pushing. That in and of itself disqualifies him from telling me anything about humans.
‍

John Campbell (41:16):

Yeah. I'm glad we talked through this. This has been fun and interesting for me to think about it more.
‍

Alicia Campbell (41:20):

Yeah. The oracle, I'm not going to be able to get over because immediately I had the matrix and I thought of Neo and everything. I'm like, oh my gosh. Well, it is wild where we are now from where that came out. Well, thank you, John, for joining us and talking about this. Hopefully everybody finds it fascinating. It's a wild development in the AI sphere, although I think one that we have pretty good handle on, it sounds like that it just isn't for trial lawyers. The summarizing part is the jury part is not. So thank you for coming. Nick, thanks for hosting again because you two guys, with your research on it, had quite a conversation. I learned a ton just sitting here listening because I haven't done as much of a deep dive as you two have in all of this. So thank you guys for coming and onto the next one.

(42:06):

We got a bunch lined up, so stay tuned for the Fred Files. I know we were off for the summer, but we're back at it. We're going to have a lot of new people and it sounds like we're going to be doing some stuff at JuryBall Vegas. So if you haven't signed up to come to JuryBall Vegas, make sure you do come out and see us. We have a lot of new topics, a lot of new presenters. We're going to have some mediators there talking about how they're implement. We're bringing them over to the side where it's better to just know something about the case than tell us you do and then not really know anything about the case. And we're getting more and more mediators to be informed when they're mediating. And then we've got some cases that we need to talk about where some new tacts are being taken by defense lawyers in terms of Amazon.

(42:45):

So show up, come on and see us. And then yeah, we'll have LawPods there. So we're going to be doing some podcasts for the Fred Files there, doing some recordings. So come and talk, come and see us. And in the meantime, have a wonderful day. Thank you guys so much.
‍

Nick Schweitzer (42:58):

Bye everyone.
‍

Alicia Campbell (42:58):

Thanks.
‍

Voice over (43:02):

Thank you for listening to the Fred Files. If you found value in today's discussion, please subscribe and share this episode with your colleagues. To explore how Fred can transform your case preparation, visit us at focuswithfred.com. Produced and Powered by LawPods.

‍