Online Legal Research: is there an affirmative duty to use more than one research platform?

Earlier this week, Robert Ambrogi posted Turns Out Legal Research Services Vary Widely in Results.  Ambrogi, one of the leading commentators on legal technology, wrote:

  • “Call me naive, but I would have thought that entering the identical search query on, say, both Westlaw and Lexis Advance would return fairly similar results, at least among the cases ranked highest for relevance. After all, shouldn’t the cases that are most relevant to the query be largely the same, regardless of the research platform?”

Then, he added:

  • “Turns out, the results they deliver vary widely — not just between Westlaw and Lexis Advance, but among several legal research platforms. In fact, in a comparison of six leading research platforms — Casetext, Fastcase, Google Scholar, Lexis Advance, Ravel and Westlaw — there was hardly any overlap in the cases that appeared in the top-10 results returned by each database.”

Ambrogi’s post referred to Susan Nevelow Mart’s research paper The Algorithm as a Human Artifact: Implications for Legal {Re}search.   Mart is the Director of the Law Library and an Associate Professor at the University of Colorado Law School.

In this column that he wrote for Above The Law, Ambrogi dove deeper in Professor Mart’s findings.  Before I talk about the findings, I want to go back to my post Are Robots Nonlawyer Assistants.  

In my post, I suggested that lawyers who use artificial intelligence to perform “mundane legal tasks” might have an affirmative duty under the Rules of Professional Conduct “to have some sort of understanding of the coder’s qualifications.”  Well, as it turns out, a similar notion underpins Professor Mart’s research.

As Ambrogi reports, several years ago, a senior VP at Westlaw informed Professor Mart that the company’s “algorithms are created by humans.”  Mart, then, theorized that the choices that a human makes in creating an algorithm will necessarily influence the results delivered by the algorithm.  In other words, that the coder’s biases & assumptions will find their way into the algorithm and impact the results.  She set out to study her hypothesis.

Mart’s findings are eye-opening.  Using the same query across 6 providers – Casetext, Fastcase, Google Scholar, LexisAdvance, Ravel and Westlaw – she found that among the top 10 cases returned by each:

  • on average, 40% of the cases were returned by only 1 provider;
  • 7% of the cases were returned by all 6 providers.

I could go on & on.  Here’s the upshot, in an excerpt of the abstract from Professor Mart’s paper:

  • When legal researchers search in online databases for the information they need to solve a legal problem, they need to remember that the algorithms that are returning results to them were designed by humans. The world of legal research is a human-constructed world, and the biases and assumptions the teams of humans that construct the online world bring to the task are imported into the systems we use for research. This article takes a look at what happens when six different teams of humans set out to solve the same problem: how to return results relevant to a searcher’s query in a case database. When comparing the top ten results for the same search entered into the same jurisdictional case database in Casetext, Fastcase, Google Scholar, Lexis Advance, Ravel, and Westlaw, the results are a remarkable testament to the variability of human problem solving. There is hardly any overlap in the cases that appear in the top ten results returned by each database. An average of forty percent of the cases were unique to one database, and only about 7% of the cases were returned in search results in all six databases. It is fair to say that each different set of engineers brought very different biases and assumptions to the creation of each search algorithm. One of the most surprising results was the clustering among the databases in terms of the percentage of relevant results. The oldest database providers, Westlaw and Lexis, had the highest percentages of relevant results, at 67% and 57%, respectively. The newer legal database providers, Fastcase, Google Scholar, Casetext, and Ravel, were also clustered together at a lower relevance rate, returning approximately 40% relevant results.

Most importantly, here’s the ethics hook:  Rules 1.1 & 1.3 require lawyers to provide competent & diligent representation. Knowing that results vary widely by provider, do Rules 1.1 and 1.3 require lawyers to use more than one provider when conducting online legal research?

Although I’ve not yet had my daily requirement of coffee, my initial reaction is that it’d be much easier to argue “yes” than to argue “no.”  Actually, the real answer might be that it’s neither competent nor diligent for a lawyer to limit research to the first 10 results to a single query.

Indeed, in the abstract to her paper, Professor Mart notes:

  • “Legal research has always been an endeavor that required redundancy in searching; one resource does not usually provide a full answer, just as one search will not provide every necessary result. The study clearly demonstrates that the need for redundancy in searches and resources has not faded with the rise of the algorithm. From the law professor seeking to set up a corpus of cases to study, the trial lawyer seeking that one elusive case, the legal research professor showing students the limitations of algorithms, researchers who want full results will need to mine multiple resources with multiple searches.”

Anyhow, I was excited to post this, but now I can’t think of a creative way to wrap it up or to make a point.  I guess my point is this: know that online legal research services aren’t perfect.

Finally, maybe Professor Mart’s findings are a new twist on something that’s been going on forever.   I’m reminded of thinking “what the _____?” when I pulled a case that did not “follow” the case that I’d been thrilled to find, even though Shepard’s had promised me (with an “f”) that it would.  The human who coded it was, in fact, only human.



Data Shows Significant Drop in Top LSAT Takers Applying to Law School

Paul Caron is the Dean of Pepperdine University’s School of Law.  He’s the editor of the popular TaxProf Blog.

Last week, Dean Caron blogged on new data that shows a significant decrease in the proportion of law school applications filed by those who score the highest on the LSAT.  Both the ABA Journal and Above The Law have more on Dean Caron’s post.

The key takeaway from the blog:

                                                              Percentage of Applications to Law School

LSAT Score                                          2010                                       2017

160 or >                                               40.8%                                    26.4%

150-159                                                45%                                        38.7%

< 150                                                     14.2%                                    34.9%     


These stories come on the heels of the ABA Journal’s post on a study in which two Pepperdine law professors linked low scores on the bar exam with disciplinary action.   The study is here.

I’ve been with the Professional Responsibility Program since 1998, and was the chief disciplinary prosecutor from 2000-2012.  We never once asked for an attorney’s bar exam score while investigating or prosecuting the attorney.  I wonder if the PRP and BBE will soon be asked to release the bar exam scores of attorneys who are (or have been) disciplined.




PRP Discusses Bias & Discrimination

The Professional Responsibility Program (PRP) held its Annual Meeting on May 31, 2017.  The meeting took place at Burlington’s Hotel Vermont.  Chief Justice Paul Reiber joined the Board, members of the PRP’s hearing & assistance panels, PRP staff, and several invited guests for a day of seminars and discussion.

The morning’s first seminar used the proposed amendment to Rule 8.4(g) as a launching point into a discussion of bias, discrimination, and legal ethics.  The audience heard from a fantastic panel of Vermont lawyers:

  • Karen Richards, Executive Director of the Vermont Human Rights Commission,
  • Jay Diaz, Staff Attorney at the ACLU of Vermont, and,
  • Dan Maguire, President-Elect of the Vermont Bar Association’s Board of Managers

Using real-life experiences and studies on racial & implicit bias, the panel challenged the audience to consider:

  • the biases that can influence hearing & assistance panel members as they sit on cases;
  • the biases that can influences lawyers, witnesses, judges, and jurors; and,
  • whether Rules 1.1 and 1.3 impose a duty to advise a client on the biases that can influence lawyers, witnesses, judges, and jurors involved in the client’s matter.

After hearing from the panel, many in the audience urged the Board to support the proposed amendment to Rule 8.4 and to commit the PRP to continue as part of the larger effort to educate on the topic of implicit bias.

Karen, Jay, and Dan – thank you so much for sharing your time, thoughts, and expertise with members of the PRP!


So You Want To Store Client Data in the Cloud….

. . . you should! Odd are it’ll make your law practice more efficient, which will help both you and your clients.

With the June 30 deadline to report CLE compliance, I’m asked to present at a lot of CLEs in May and June.  This year, several folks have asked me to talk about the ethics associated with storing client data in the cloud.

I will do as asked. Reluctantly.

Last November, I posted a blog in which I expressed my hope that I’d done my last seminar on the ethics of storing information in the cloud.  I think it’s time we move beyond “can I use the cloud?” to figuring out whether the cloud works for you & your firm and, if so, which vendor to choose.

Since my hope has not yet been realized, I’m re-posting my post. Two words to remember: “Reasonable Precautions.”


The Cloud:  What are Reasonable Precautions?

Last Friday, I presented a CLE for the Rutland County Bar Association. My assigned topic: the ethics of storing client information in the cloud.  I started by saying that I hoped it was my final seminar on the topic.  I was serious.

Let’s walk through this.

In general, a lawyer has a duty not to disclose information relating to the representation of a client absent client consent.  See, Rule 1.6.  A lawyer also has a duty to keep client property safe.  See,Rule 1.15.

I view the cloud as the latest in a long line of different places to store information.  In that sense, the cloud is not different than manila folders, boxes, offices, attics, basements, barns, file cabinets, file cabinets with locks, storage facilities, hard drives, floppy disks, CDs, and thumb drives.

No matter where a lawyer stores client information, a lawyer must act competently to protect the information against inadvertent or unauthorized disclosure. See, Rule 1.6, Comment [16].  When transmitting client information, a lawyer must take reasonable precautions to prevent the information from coming into the hands of unintended recipients.  Rule 1.6, Comment [17].

So, think about cloud storage like this:  client information is electronically transmitted to a place where it will be kept.  Thus, a lawyer must take reasonable precautions to protect client information both while it is in transit and while it is at rest.

In fact, that’s almost exactly what the VBA’s Professional Responsibility Committee said – SIX YEARS AGO when it issued Advisory Ethics Opinion 2010-06.  Here’s the digest of the opinion:

  • “Vermont attorneys can utilize Software as a Service in connection with confidential client information, property, and communications, including for storage, processing, transmission, and calendaring of such materials, as long as they take reasonable precautions to protect the confidentiality of and to ensure access to these materials.”

(Aside: for anyone wondering why I included an advisory opinion about “Software as a Service” in a post on cloud computing, I remind you that Rule 1.0’s duty of competence includes tech competence.)

The question I hear most often is this:  “what are reasonable precautions?”  In Rutland, I suggested to the audience that they already know the answer, if only by treating the cloud as if it were a storage facility out on Old County Road. Some questions you might ask when considering that facility:

  • who do you let into this facility?
  • do you require a passcode or badge for the gate?
  • are there locks on the individual units?
  • who besides me has a key or knows the combination?
  • can i get into my unit whenever i want to?
  • what happens to my files if I don’t pay or if you go out of business?

Indeed, take a look at page 6 of the VBA Opinion.  The Committee suggested some of those exact questions when considering a cloud vendor.

Or, take a look at this post from Robert Ambrogi.  He writes that “[s]ome basic questions to ask of a cloud vendor, distilled from various ethics opinions, include:

  • Is it a solid company with a good reputation and record?
  • Can you get access to your data whenever you want, without restrictions?
  • If your service is terminated – by you or by the company – can you retrieve your data?
  • Does it allow use of advanced password protocols and two-step verification?
  • What are its internal policies regarding employee and third-party access to your data?
  • Is your data encrypted both while in transit and while at rest on the company’s servers?
  • How is your data backed up?
  • What security protections are in place at the data centers the company uses?”

Finally, remember that asking the questions isn’t enough.  You need to understand the answers or find someone who does.  For example, imagine this:

  • You:   Will my data be encrypted in transmission and at rest?
  • Vendor:  Yes.  In transmission, we use a BTTF Flux Capacitor.  At rest, we use the latest cloaking technology from Romii.
  • You.  Sounds awesome. Sign me up.

Umm, no.  You just signed up to star in the next entry in Was That Wrong.

In conclusion, you may store client information in the cloud so long as you take reasonable precautions.  This entry includes links that will help you determine what “reasonable precautions” are.  Don’t fear the cloud, but know what you don’t know.

Speaking of which, info on the BTTF Flux Capacitor is HERE. And, for more on Romii cloaking technology, go HERE.


Cloud for Lawyers

Are Robots Nonlawyer Assistants?

Seyfarth Shaw is a law firm with an impressive list of accolades.  And, now, the firm appears poised to be the first major law firm to use robots to handle tasks presently performed by lawyers.

In a joint press release with a company called Blue Prism, Seyfarth Shaw announced:

  • “We’re excited about the opportunity this creates to free our lawyers from some of the more mundane legal tasks so they can focus on helping our clients solve their most complex business issues,” explained Seyfarth’s chair emeritus Stephen Poor. “In testing various use cases, we’ve already seen how Blue Prism’s RPA software can help us create exponential gains in productivity, and we’ve only begun to scratch the surface of possibilities.”

The ABA Journal has the full story here.

A phrase stood out:  “[w]e’re excited about the opportunity this creates to free our lawyers from some of the more mundane legal tasks . . ..”

So, it looks to me as if robots will be performing “mundane legal tasks.”

I’m not the least bit surprised.  But, from a regulatory perspective, what if the robot gets it wrong?

In Vermont, Rules 5.3(a) & (b) impose responsibilities regarding nonlawyer assistants.  Rule 5.3(c) holds a lawyer ethically liable for the conduct of a nonlawyer assistant if the lawyer orders or ratifies it, or if the lawyer has knowledge of a nonlawyer assistant’s conduct and fails to take reasonable remedial action at a time when the consequences can be avoided or mitigated.

As I’ve often said, Rule 1.1’s duty of competence includes tech competence.  Read together, do Rules 1.1 and 5.3 require lawyers who use robots to have some sort of understanding of the coder’s qualifications?  Perhaps we will eventually treat the purchase of robots as we do the selection of a cloud vendor and hold that “a lawyer must take reasonable precautions in choosing a robot that will perform mundane legal tasks.”

Even beyond choosing the robot, is there a duty to “trust but verify” the robot’s work?  I have no idea what “mundane legal tasks” the robots will be doing.  However, absent random quality assurance checks, it’s conceivable that the robots could get a task wrong for quite a period of time before anyone realizes it.  Not only that, I’d assume that a mistake would result from a programing error and, therefore, could be repeated over & over & over again.  Or, will this have been addressed in the testing phase?

The profession’s eventual replacement of humans with machines intrigues me, even if only from an ethics perspective.  Are machines burdened by notions of loyalty?  If not, will the conflict of interest rules apply to robots?

In any event, this is only the beginning.  As the press release goes on to state:

  • “Blue Prism provides an anchor around which we can refine and test the types of robotics that immediately make our lawyers better and faster,” said Byong Kim, director of technology innovations, SeyfarthLean Consulting. “At its core, this is about arming lawyers with the best technology, and software robots are the latest evolution.




Service by Social Media

I’m at the mid-winter meeting of the National Organization of Bar Counsel. The program starts later today and I expect to learn lots of  blog-worthy info.

In the meantime, here’s something related to tech competence.

Earlier today, the ABA Legal Technology Resource Center shared a link from the ABA’sLaw Practice Group. The link, which is here, is to an update on the evolving area of using social media to effect service.

Competence includes tech competence. img_1938

Tips for Choosing a Practice Management System

Most of you know that when it comes to legal tech, I highly recommend Robert Ambrogi’s Law Sites Blog.  Ambrogi also writes a This Week In Legal Tech column for Above The Law.

Here’s the most recent column: 6 Questions To Ask Before Selecting A Practice Management Platform.

Read it.

A summary of the 6 questions:

  1. Do you want a cloud platform or a platform installed on site?
  2. How much do you want to pay?
  3. Does the system comply with security requirements and obligations under the Rules of Professional Conduct?
  4. Does it have the basic features that you need?
  5. Does it have the advanced features that you need?
  6. Does it feel right when you try it?

Again, read the article.

For part 2 of question 3, my view is that a lawyer’s obligation under the Rules of Professional Conduct is to take reasonable precautions to protect client data, whether the data is in transmission or at rest.  What are reasonable precautions?  I addressed that question HERE.

Still drinking coffee this morning?  You’ve got time to try this week’s legal ethics quiz before you hit the trails or slopes.



Clients: Ease Their Experience

I’m not sure how to tie this to legal ethics, but I wanted to share it.  So, I’ll say this: Rule 1.1 of the Rules of Professional Conduct requires lawyers to provide competent representation.  I’m about to share with you an article that, at some level, suggests it’s possible that clients perceive competent representation as including a smooth ride along the way.  By “smooth ride,” I’m not talking results.  Rather, I’m referring to the ease of the client experience.

The post is HERE. It’s by Jack Newton, founder and CEO of Clio.  It appears on the ABA Journal’s site.

Referring to a survey conducted by CEB, Inc., of more than 75,000 consumers, Newton writes:

  • “CEB found that modern consumers are not seeking exceptional customer experiences. Instead, they prefer effortless experiences.”

From there, Newton referenced Uber and Amazon, noting that:

  • “For examples of a smooth experience, we can look to innovators of the 21st century. These companies have disrupted deep-pocketed incumbents by delivering truly effortless experiences.”

Check out the post.  Some of you might find it helpful.  You might ask yourself, “self, am I delivering an effortless experience to my clients?”  Indeed, as the Judiciary, we should be asking the same question: are we delivering an effortless experience to our consumers? If not, how can we make it closer to the type of experience that consumers expect?

Newton’s post also includes fascinating thoughts on data.  I’ve always been struck how resistant the legal profession is to using data. With respect to collecting fees, Newton notes that the:

  • “data paints a bleak picture: Out of an eight-hour workday, the average firm collects payment on only 1.5 hours of billable time. These unit economics would be devastating to almost any industry, and they help explain why—despite charging an average of $232 per billable hour—the average small-to-midsize firm struggles to make ends meet.”

Imagine representing a restraunt owner who tells you that of every 8 meals served, she only collects payment for 1.5?

Again, take a look at Newton’s post. It includes some excellent tips on using data to make your practice more efficient.

I agree with Newton’s conclusion:

  • “The winning law firms of the future will take a page from the disrupter’s playbook: Deliver truly effortless customer experiences while advancing a ruthlessly data-driven culture of continuous improvement internally. Combined, these two forces will reshape the face of legal services.”

In the end, I suspect it will be the lawyers who figure out how to make the client experience effortless who see a significant increase in the number of billable hours collected.  In that sense, competent representation will not only benefit your clients, it will benefit you.


CC & Reply-All: Is Bcc the Answer?

This week’s post on issues that can arise when a lawyer copies a client on an e-mail sent to opposing counsel generated signficant discussion.

In the post, I referred to this advisory opinion from the New York State Bar Association. The opinion suggests that a more prudent course of action is for a lawyer to send the e-mail to opposing counsel, then forward it to the client from the lawyer’s “sent” items.

Several readers suggested that a “bcc” to the client is simpler and avoids any concerns about opposing counsel replying directly to the client.


A “bcc” to the client certainly prevents opposing counsel from concluding that you, the sender, have consented to opposing counsel having direct contact with your client.  But, do you know what happens if the blind-copied client uses “reply-all?”

I’ve tested this twice. Once at CLE in Rutland, and again yesterday with two co-workers.  Each time, we “proved” the result.  Still, I suspect many of you will run the test yourselves.

I work with Deb and Brandy.  For purposes of the test, pretend that I represent Brandy and that Deb is opposing counsel.  Yesterday, I sent an e-mail to Deb and blind copied Brandy. In other words, I sent an e-mail to opposing counsel and blind copied my client.

I asked each to try to “reply-all.”

  • Deb’s reply went only to me, not to Brandy, my client. Indeed, when Deb clicked “reply-all,” the only address that appeared in the window was mine.  So, yes, the bcc to my client prevented opposing counsel from replying to my client.

My client was another story.

  • Brandy replied to all.  By “all”, her reply went to me AND to Deb.  That’s right: even though Brandy had been bcc’ed on my email to Deb, Brandy was able to “reply-all” to me and to Deb.

Now, I know lawyers love blanket statements. I’m not making one.  That is, I am not saying “a lawyer violates the Rules of Professional Conduct by blind copying a client on an e-mail to opposing counsel.”  Here’s what I’m saying: it’s not the magic bullet you might think it is.

Let’s say that my e-mail to Deb indicated that Brandy would settle a civil claim for $100,000, but nothing less.  Imagine that Brandy, intending to reply only to me, accidentally used “reply-all” to write “Awesome! Do you think it will work? Even if it doesn’t, no big deal. I’ve said all along that I’d take $33,000 in a heart beat. By the way, what happens if they find out I was texting when it happened?”

I can hear you now.  “Mike, what are the odds?”  Well, here’s an excerpt from the NYSBA advisory opinion:

  • “12. Although sending the client a ‘bcc:’ may initially avoid the problem of disclosing the client’s email address, it raises other problems if the client mistakenly responds to the e-mail by hitting “reply all.”  For example, if the inquirer and opposing counsel are communicating about a possible settlement of litigation,  the inquirer bccs his or her client, and the client hits ‘reply all’ when commenting on the proposal, the client may inadvertently disclose to opposing counsel confidential information otherwise protected by Rule 1.6.  See Charm v. Kohn, 27 Mass L. Rep. 421, 2010 (Mass. Super. Sept. 30, 2010) (stating that blind copying a client on lawyer’s email to adversary “gave rise to the foreseeable risk” that client would respond without ‘tak[ing] careful note of the list of addressees to which he directed his reply’).”

So, yes: a bcc to a client eliminates the risk that opposing counsel will conclude that you’ve consented to opposing counsel communicating directly with your client.  However, it does not eliminate the risk that your client accidentally discloses confidential and privileged information in a “reply-all.” Indeed, per the Massachusetts case, it creates a “foreseeable risk” that the client will do exactly that.  The opinion is HERE.

If you bcc a client on an e-mail to opposing counsel, make sure the client understands that “reply-all” will not be for your eyes only.

For your eyes only.jpg



Moneyball: Are Lawyers Required to Scout Judges?

As many of you know, I coached high school basketball for a long, long time.  Contrary to popular belief, high school coaches do far more than cut your kid, bench your kid, or play your kid, but without letting him take every shot. One of the things that coaches do is scout opponents.

Scouting entails going to a future opponent’s game and studying the opponent.  What defense does the team play?  If the coach calls timeout, what play is the team likely to run next? Which players do we force to dribble to the left – because they can’t? Which players are such poor shooters that we don’t need to worry about guarding them? Stuff like that.

Coaches also scout referees.  Which refs call hand-checking?  Which refs don’t care about physical play in the lane?  Which refs call moving screens?  Stuff like that.

Knowing “stuff like that” is considered MINIMAL competence in the coaching world.

And now it may have (officially) entered the attorney ethics world.

Again, lawyers must provide clients with competent and diligent representation. Many of you have informally scouted judges.  You know which judges are inclined to stick to the letter of the law.  You know which judges are inclinded to respond to “fairness” arguments.  You know which judges are likely to grant motions to suppress, and, you know which judges aren’t.  Stuff like that.

But even if you don’t know stuff like that, I’m not aware of anything that says you’re less than diligent or competent.  That might be changing.

Earlier this week, Robert Ambrogi posted this entry to his fabulous Law Sites blog.  Here’s a quick summary: Ravel Law launched a service that allows lawyers to scout judges. It’s called “Court Analytics.”  Describing the service,  Ambrogi notes:

  • “you can see, for example, a court’s most-cited opinions and judges, as well as the opinions and courts it most frequently cites. It also lets you identify how courts and judges have ruled in the past on particular issues or motion types.”

Ambrogi added:

  • “How would an attorney use these analytics? I put that question last week to Daniel Lewis, the co-founder and chief executive officer of Ravel Law. He described two primary use cases:
    • Forum comparison. If an attorney is forum shopping and wants to compare how different jurisdictions have dealt with a particular issue or motion.
    • Argument crafting. If an attorney is arguing a matter to a court, the argument can be made more persuasive by knowing which authorities that court or judge finds most persuasive.

    ‘Attorneys will be able to use it to see how a court has dealt with cases on a particular topic or motion, what they key cases are they should know about, and the particular rules, standards and language that are most important in that venue,’ Lewis said.”

Frankly, I’m surprised it has taken so long for analytics to arrive in the legal profession.  As any baseball or movie fan knows, using analytics is now considered minimal competence in baseball. Show me an industry, I’ll show you an industry in which analyzing data is considered minimal competence.

Except, until now, the law.

It makes me wonder: as the data that drives Ravel’s “Court Analytics” app becomes more freely available, will the duty of competence include a duty to make use of that data? It will, right?  After all, what will your response be when asked “why didn’t you?”

By the way, for those of you thinking “there’s no way that this impacts Vermont,” think again.  As Ambrogi notes:

  • “Thanks to Ravel’s digitization partnership with Harvard Law School, its case law collection includes all federal and state courts.”