Hard numbers

As I’ve mentioned before (including yesterday), everybody seems to be beating up the legal job market these days.  The American Bar Association apparently decided that it was time to inject some actual numbers into the discussion:

[Most prior discussion has] been based in great part on the tools of journalism: anecdote, instinct and the oft-competing wisdom of any experts we can find.

With this issue, however, the ABA Journal is offering our readers a new—and we believe different—view of the business and the profession.

We’ve teamed up with a nationally recognized expert on trends in the legal profession, William D. Henderson of the Center on the Global Legal Profession at Indiana University’s Maurer School of Law. We asked Henderson, a pioneer in the empirical study of the legal industry, to identify and map the movements of jobs and money.

There’s a separate page that allows county-by-county data searching.

Here’s the thing:  based on my look at the publicly available U.S. Bureau of Labor Statistics data, underlying the ABA’s “report”, I’m not quite sure what the ABA has added to the discussion here.  Sure, they’ve generated some colorful graphs and county-by-county maps.  But as far as I can tell, all (and I do mean all) of this data has been around since at least May 14, 2010.  And it’s not like the ABA has done much analysis here; they’ve basically just sorted the size of salaries out by metro region and announced a few “surprises”.

Even more problematically, I’m not sure there are many clear takeaways due to the inherent shortcomings of this data.  Per the bottom of the article’s main page:

The [U.S. Bureau of Labor Statistics] data are a representative sample of employed lawyers. The sample includes lawyers employed in law firms, state and local government, federal government, in-house lawyers in businesses, and nonprofits. Lawyers, as defined by the BLS classification (SOC), “represent clients in criminal and civil litigation and other legal proceedings, draw up legal documents, and manage or advise clients on legal transactions. May specialize in a single area or may practice broadly in many areas of law.” Equity partners and solo practitioners are not included in the survey. [emphasis added]

In other words:

  1. This data leaves out solo practitioners — fully 35% of all lawyers according to Harvard Law School’s research.  Analysis:  these salary numbers skew high.  (I suppose the lack of focus on solos isn’t too surprising since only about 7% of all solos belong to the ABA anyway.)
  2. This data only applies to employed lawyers.  Analysis:  This article tells us nothing about the marginal earning prospects of unemployed lawyers, including recently graduated J.D.’s who are “temporarily” employed in other industries (e.g., as servers in restaurants).

I get that this is “the first installment of a periodic series.”  But come on, ABA.  It’s more than a little disingenuous to claim that “the ABA Journal is offering our readers a new—and we believe different—view of the business and the profession” by “identify[ing] and map[ing] the movements of jobs and money” when you’re simply re-publishing eight month old government data with an arguably misleading slant and without substantive analysis.

Fighting innumeracy and “proofiness”

A new book by journalist Charles Seife examines how figures and statistics are poorly used in public debates. I like his idea of “proofiness” which seems similar to the concept of “truthiness.” Here are some of the types of bad statistics he points out:

Falsifying numbers is the crudest form of proofiness. Seife lays out a rogues’ gallery of more subtle deceptions. “Potemkin numbers” are phony statistics based on erroneous or nonexistent calculations. Justice Antonin Scalia’s assertion that only 0.027 percent of convicted felons are wrongly imprisoned was a Potemkin number derived from a prosecutor’s back-of-the-envelope estimate; more careful studies suggest the rate might be between 3 and 5 percent.

“Disestimation” involves ascribing too much meaning to a measurement, relative to the uncertainties and errors inherent in it. In the most provocative and detailed part of the book, Seife analyzes the recounting process in the astonishingly close 2008 Minnesota Senate race between Norm Coleman and Al Franken. The winner, he claims, should have been decided by a coin flip; anything else is disestimation, considering that the observed errors in counting the votes were always much larger than the number of votes (roughly 200 to 300) separating the two candidates.

“Comparing apples and oranges” is another perennial favorite. The conservative Blue Dog Democrats indulged in it when they accused the Bush administration of borrowing more money from foreign governments in four years than had all the previous administrations in our nation’s history, combined. True enough, but only if one conveniently forgets to correct for inflation.

Books like these are needed in our society as politicians often debate through numbers. Without a proper understanding of who is using these numbers, where they come from, and what they mean, the public will have difficulty understanding what is going on. (And this may be the aim of politicians.)

(Based on this review, his arguments and concepts seem similar to those of sociologist Joel Best.)