Tag Archives: ABA

Credibility, statistics, and the legal profession

Posted on by

Elie Mystal at Above the Law has this take on a recent story involving credibility, statistics, and the legal profession:

This week, the law schools at Columbia, NYU, and Fordham have come under fire for their allegedly inflated employment statistics. A story in the New York Post specifically called out the top New York-area law schools for shady reporting of graduate outcomes when it comes to graduates employed by the schools….

I want to take a step back and look at what we’re really fighting about here: some of the best law schools in New York City have put out a statistic about how many graduates get jobs, and the New York Post and a bunch of other people immediately called “bulls**t.” Think about that. Even if the law schools can somehow convince people that, technically, their published information isn’t riddled with lies, we’re living in a world where such data can be assumed to be false absent a long and detailed explanation and discussion from the law schools. When somebody notices a discrepancy between a school’s numbers and what’s in the newspaper, we assume the school was full of crap, not that the newspaper got it wrong.

I suppose this isn’t very flattering to either newspapers or law schools.  Perhaps Americans now trust journalists more than lawyers (or at least legal educators)?

Commenting a few months ago on a scandal within academic sociology, Brian suggested several approaches to dealing with uncertain statistics:

This reminds me of Joel Best’s recommendations regarding dealing with statistics. One common option is to simply trust all statistics. Numbers look authoritative, often come from experts, and they can be overwhelming. Just accepting them can be easy. At the other pole is the common option of saying that all statistics are simply interpretation and are manipulated so we can’t trust any of them. No numbers are trustworthy. Neither approaches are good options but they are relatively easy options. The better route to go when dealing with scientific studies is to have the basic skills necessary to understand whether they are good studies or not and how the process of science works [emphasis added].

Brian’s point is a good one.  Unfortunately, it’s not possible to implement his “third way” here because the root problem is the lack of raw information rather than the inability to duplicate experimental/study results.  The question is not, in the theoretical abstract:  how many law students will get jobs when the economy is in condition X?  The question is rather:  as a matter of historical fact, how many 2010 law school graduates (or 2011, or 2009, or whatever) actually had jobs by date Y?

Elie faults the American Bar Association, which oversees and accredits law schools, for the current disaster of unreliable data:

The ABA is supposed to represent lawyers and law schools to the public. It’s supposed to relegate them so that the public can trust that moral and ethical standards are being upheld and enforced. And on that scale, the ABA has been an unmitigated failure. It’s done a disservice to all law schools. Nobody can trust any law school because the ABA has failed to impose effective oversight over all of them.

That’s tragic. A society is supposed to be proud of its institutions of higher learning, but the ABA has robbed us of that pride in our nation’s law schools. We no longer get to feel like our justice system is populated by people trained to the highest ethical standards, because we can’t even trust our law schools to tell us the truth about how many people got hired.

If the numbers published by law schools under the oversight of the ABA are unreliable, it goes without saying that it’s very difficult to derive these numbers through other means, especially in a form that allows for legitimate comparisons between schools and over time.  There are workarounds, of course, like journalistic attempts to compile and/or verify employment statistics independently of the law schools.  But those are obviously imperfect solutions, as The Economist recently noted in its analysis of the (surprisingly analogous) problem of Argentinan inflation statistics:

Since 2007 Argentina’s government has published inflation figures that almost nobody believes. These show prices as having risen by between 5% and 11% a year. Independent economists, provincial statistical offices and surveys of inflation expectations have all put the rate at more than double the official number (see article). The government has often granted unions pay rises of that order….

We [The Economist] hope that we can soon revert to an official consumer-price index for Argentina. That would require INDEC to be run by independent statisticians working unhindered. Until then, readers are better served by a credible unofficial figure than a bogus official one.

Unfortunately, for the foreseeable future, I think we’re going to need to start seeing a lot more credible unofficial figures out there, both for Argentinian inflation and for law school placement statistics.

Hard numbers

Posted on by

As I’ve mentioned before (including yesterday), everybody seems to be beating up the legal job market these days.  The American Bar Association apparently decided that it was time to inject some actual numbers into the discussion:

[Most prior discussion has] been based in great part on the tools of journalism: anecdote, instinct and the oft-competing wisdom of any experts we can find.

With this issue, however, the ABA Journal is offering our readers a new—and we believe different—view of the business and the profession.

We’ve teamed up with a nationally recognized expert on trends in the legal profession, William D. Henderson of the Center on the Global Legal Profession at Indiana University’s Maurer School of Law. We asked Henderson, a pioneer in the empirical study of the legal industry, to identify and map the movements of jobs and money.

There’s a separate page that allows county-by-county data searching.

Here’s the thing:  based on my look at the publicly available U.S. Bureau of Labor Statistics data, underlying the ABA’s “report”, I’m not quite sure what the ABA has added to the discussion here.  Sure, they’ve generated some colorful graphs and county-by-county maps.  But as far as I can tell, all (and I do mean all) of this data has been around since at least May 14, 2010.  And it’s not like the ABA has done much analysis here; they’ve basically just sorted the size of salaries out by metro region and announced a few “surprises”.

Even more problematically, I’m not sure there are many clear takeaways due to the inherent shortcomings of this data.  Per the bottom of the article’s main page:

The [U.S. Bureau of Labor Statistics] data are a representative sample of employed lawyers. The sample includes lawyers employed in law firms, state and local government, federal government, in-house lawyers in businesses, and nonprofits. Lawyers, as defined by the BLS classification (SOC), “represent clients in criminal and civil litigation and other legal proceedings, draw up legal documents, and manage or advise clients on legal transactions. May specialize in a single area or may practice broadly in many areas of law.” Equity partners and solo practitioners are not included in the survey. [emphasis added]

In other words:

  1. This data leaves out solo practitioners — fully 35% of all lawyers according to Harvard Law School’s research.  Analysis:  these salary numbers skew high.  (I suppose the lack of focus on solos isn’t too surprising since only about 7% of all solos belong to the ABA anyway.)
  2. This data only applies to employed lawyers.  Analysis:  This article tells us nothing about the marginal earning prospects of unemployed lawyers, including recently graduated J.D.’s who are “temporarily” employed in other industries (e.g., as servers in restaurants).

I get that this is “the first installment of a periodic series.”  But come on, ABA.  It’s more than a little disingenuous to claim that “the ABA Journal is offering our readers a new—and we believe different—view of the business and the profession” by “identify[ing] and map[ing] the movements of jobs and money” when you’re simply re-publishing eight month old government data with an arguably misleading slant and without substantive analysis.

ABA wants free trade in legal services?

Posted on by

The International Business Times is reporting that American Bar Association President Stephen N. Zack is lobbying India to refrain from shutting U.S. lawyers out of the Indian legal market:

Currently, U.S. lawyers are allowed to travel to India on an “in-and-out basis” to advise their clients on non-Indian aspects of law. That “status quo” should be maintained as the [Bar Council of India] considers the broader issue of whether to allow the practice of law by foreign law firms in India, Zack said….”The ABA believes that allowing such activities is critical not only for the mutual benefit of the legal practitioners in both countries,” [Zack’s] letter said, “but also for fostering the vital and already close relationship between India and the United States and to promote the robust growth of trade and investments between our two countries. Allowing such activities is also essential in making India a preferred venue for international arbitration proceedings.”

This is a huge issue, and only going to get bigger in the coming years.  In most countries, including the U.S. and India, the legal profession is highly regulated and heavily skewed toward protectionism (i.e., preserving a pre-globalization status quo).  For example, in order to “practice law” in the U.S., one must generally graduate from college, attend law school for 3 years, and pass a state-specific bar exam.  Other countries have similarly stringent requirements.  Obviously, most people who have been through the trouble (and expense) of this process are vehemently opposed to competition from anyone else–including (and especially) lawyers licensed in other countries.

Which is what makes the ABA president’s statements so interesting.  Supposedly, U.S. lawyers currently provide Indian businesses with “consultancy legal services” (to use the article’s phrase) rather than “practice law” (which is the magic phrase to denote what one cannot do without an official license in a given state/country).  However, such verbal formulations are notoriously vague, and everyone who argues over their precise meanings are lawyers with a vested interest in either (1) expanding their own market for legal services or (2) keeping new competition out.

To date, new competition has mostly been kept out, especially here in the U.S.  It will be interesting to see whether the ABA president’s recent lobbying in India represents a first step moving toward a free trade in legal services between the U.S. and India.