Tag Archives: Watson

500 to 1

Posted on by

I contemplated the effects of technological changes on law jobs several weeks ago when I posted a link to news reports about IBM’s Watson winning Jeopardy.  The New York Times has written what essentially amounts to a follow-up article, and it’s eye opening:

Quantifying the employment impact of these new technologies [that help automate the legal discovery process] is difficult. Mike Lynch, the founder of Autonomy, is convinced that “legal is a sector that will likely employ fewer, not more, people in the U.S. in the future.” He estimated that the shift from manual document discovery to e-discovery would lead to a manpower reduction in which one lawyer would suffice for work that once required 500 and that the newest generation of software, which can detect duplicates and find clusters of important documents on a particular topic, could cut the head count by another 50 percent. [emphasis added]

To be sure, 500:1 may just be the talking point of a businessman who is trying to sell his particular solution. Nonetheless, it seems clear that technology like Mr. Lynch’s is already fundamentally altering the economics of the legal profession.  We probably are headed towards a future with fewer lawyers (at least, ones performing discovery-related tasks).

What are some of the broader economic implications?  The NYTimes piece also quotes from  David H. Autor, an economics professor at the Massachusetts Institute of Technology:

“There is no reason to think that technology creates unemployment,” Professor Autor said. “Over the long run we find things for people to do. The harder question is, does changing technology always lead to better jobs? The answer is no.”

The billable value of humility

Posted on by

In a previous post, I linked to an IBM executive who claimed that Watson’s success on Jeopardy! might revolutionize the legal profession.  Gary Kasparov, the chess champion who was defeated by IBM’s Deep Blue in 1997, makes an interesting observation over at the Atlantic:

My concern about its utility, and I read they would like it to answer medical questions, is that Watson’s performance reminded me of chess computers. They play fantastically well in maybe 90% of positions, but there is a selection of positions they do not understand at all….A strong human Jeopardy! player, or a human doctor, may get the answer wrong, but he is unlikely to make a huge blunder or category error—at least not without being aware of his own doubts. We are also good at judging our own level of certainty…but I would not like to be the patient who discovers the medical equivalent of answering “Toronto” in the “US Cities” category, as Watson did.

It would be ironic (in the extreme!) if what saved the legal profession from being taken over by computers was lawyers’ own humility.  In a world where machines can process far more data far faster than human minds, perhaps our weakness is also our greatest asset:  we know our limits.

The prospect of the automated grading of essays

Posted on by

As the American public debates the exploits of Watson (and one commentator suggests it should, among other things, sort out Charlie Sheen’s problem) how about turning over grading essays to computers? There are programs in the works to make this happen:

At George Mason University Saturday, at the Fourth International Conference on Writing Research, the Educational Testing Service presented evidence that a pilot test of automated grading of freshman writing placement tests at the New Jersey Institute of Technology showed that computer programs can be trusted with the job. The NJIT results represent the first “validity testing” — in which a series of tests are conducted to make sure that the scoring was accurate — that ETS has conducted of automated grading of college students’ essays. Based on the positive results, ETS plans to sign up more colleges to grade placement tests in this way — and is already doing so.

But a writing scholar at the Massachusetts Institute of Technology presented research questioning the ETS findings, and arguing that the testing service’s formula for automated essay grading favors verbosity over originality. Further, the critique suggested that ETS was able to get good results only because it tested short answer essays with limited time for students — and an ETS official admitted that the testing service has not conducted any validity studies on longer form, and longer timed, writing.

Such programs are only as good as the algorithm and method behind it. And it sounds like this program from ETS still has some issues. The process of grading is a skill that teachers develop. Much of this can be quantified and placed into rubrics. But I would also guess that many teachers develop an intuition that helps them quickly apply these important factors to work that they read and grade.

But on a broader scale, what would happen if the right programs could be developed? Could we soon reach a point where professors and teachers would agree that a program could effectively grade writing?

Law jobs in Jeopardy

Posted on by

There’s been a lot of talk this week about Watson’s appearance on Jeopardy! — and its win.  Now, the pundits are trying to digest what the implications will be now that Watson has already been hired as a physician’s assistant.

What, specifically, does this mean for lawyers?  Robert C. Weber, a senior VP and general counsel at IBM, breaks it down for us over at over at the National Law Journal:

Imagine a new kind of legal research system that can gather much of the information you need to do your job — a digital associate, if you will. With the technology underlying Watson, called Deep QA, you could have a vast, self-contained database loaded with all of the internal and external information related to your daily tasks, whether you’re preparing for litigation, protecting intellectual property, writing contracts or negotiating an acquisition. Pose a question and, in milliseconds, Deep QA can analyze hundreds of millions of pages of content and mine them for facts and conclusions — in about the time it takes to answer a question on a quiz show.

But won’t this mean fewer jobs for lawyers?  Oh no, reassures Mr. Weber:

Deep QA won’t ever replace attorneys; after all, the essence of good lawyering is mature and sound reasoning, and there’s simply no way a machine can match the knowledge and ability to reason of a smart, well-educated and deeply experienced human being. But the technology can unquestionably extend our capabilities and help us perform better.

Humanity — I mean — lawyers win, huh?  This is great!  Where can I put Watson to work?

The technology might even come in handy, near real-time, in the courtroom. If a witness says something that doesn’t seem credible, you can have an associate check it for accuracy on the spot.

Wait a minute — I thought you said that we’ll always need lawyers?  But if using Watson/Deep QA is just as easy as running a Google search against a witness on the stand, why do you need to have an associate perform it?  Associates are expensive, or, at least, used to be.  Why not a paralegal?  Why not someone even cheaper, with even less training?  Are you sure it has to be an actual lawyer?  (Besides, Weber also tells us that “We’re pretty sure [Watson] would do quite well in a multistate bar exam!”)

Perhaps when he said Watson “won’t ever replace attorneys,” Mr. Weber meant that Watson won’t ever replace someone like himself:  a successful, established, general counsel at a Fortune 500.  You know, the sort of person who passes off his “research” to an “associate.”  Or whomever.  Or whatever.

I’m not buying it, Weber (neither is Above the Law, for whatever that’s worth).  Watson is going to put a lot of lawyers out on the street, which is precisely the conclusion that Andy Kessler comes to over at the Wall Street Journal.  In Kessler’s colorful employment taxonomy, lawyers are classified as “sponges”:

Sponges are those who earned their jobs by passing a test meant to limit supply. According to [the WSJ], 23% of U.S. workers now need a state license….All this does is legally bar others from doing the same job, so existing workers can charge more and sponge off the rest of us.

But eDiscovery is the hottest thing right now in corporate legal departments. The software scans documents and looks for important keywords and phrases, displacing lawyers and paralegals who charge hundreds of dollars per hour to read the often millions of litigation documents. Lawyers, understandably, hate eDiscovery.

We can argue whether this is a good for society overall (or not).  But come on, Weber.  Don’t say that Watson “won’t ever replace attorneys” when what you really mean is that “I personally am going to be able to keep my job.”