Sociology professor developed and used computer program for grading papers

Sociologist Ed Brant has developed and used a grading program for student papers:

Brent designed software called a SAGrader to grade student papers in a matter of seconds. The program works by analyzing sentences and paragraphs for keywords and relationships between terms. Brent believes the program can be used as a tool to save time for teachers by zeroing in on the main points of an essay and allowing teachers to rate papers for the use of language and style.

“I don’t think we want to replace humans,” Brent says in an article in Wired. “But we want to do the fun stuff, the challenging stuff. And the computer can do the tedious but necessary stuff.”

Using the software still requires work on the teacher’s part, though. To prepare the program to grade papers, a teacher must enter all of the components they expect a paper to include. Teachers also have to consider the hundreds of ways a student might address the pieces of an essay.

Interestingly, one person in the testing business argues that the biggest issue is not how well the software does at grading but whether people believe the program can do a good job:

But it’s tough to tout a product that tinkers with something many educators believe only a human can do.

“That’s the biggest obstacle for this technology,” said Frank Catalano, a senior vice president for Pearson Assessments and Testing, whose Intelligent Essay Assessor is used in middle schools and the military alike. “It’s not its accuracy. It’s not its suitability. It’s the believability that it can do the things it already can do.”

If this were used widely and becomes normal practice, it could redefine what it means to be a professor or teacher. This is not a small issue in an era where many argue that learning online or from a book could be as effective (or at least as cost-effective) compared to sending students to pricey colleges.

I wonder what percentage of sociologists would support using such grading programs in their own classrooms and throughout academic institutions.

Judging the validity of academic expertise in court

It is common in the world of academia for academics to judge the credibility of other scholars. But what happens when academics step into the courtroom and a judge assesses whether they are experts or not? Consider the case of a Canadian sociologist who was going to testify as a gangs expert:

Mark Totten, an Ottawa sociologist, has “virtually no expertise with gangs in the Greater Toronto Area,” Ontario Superior Court Justice Robert Clark said in a 27-page ruling which had been under a publication ban until the jury in a gang-related case began deliberations Wednesday.

Yet, this is the same sociologist who, in 2009, was praised by the Ontario Court of Appeal for having “extensive and impressive credentials” in the field of street gang culture…

Totten himself admitted in an interview that he “didn’t handle it very well” after wilting under cross-examination in the voir dire, a preliminary examination to determine the competency of a witness…

The last time Totten’s expertise was questioned was in 2007, when Justice Todd Archibald disallowed his “expert” witness testimony on the meaning of a teardrop tattoo on the cheek of an accused killer, Warren Abbey…

On his website, Totten’s list of degrees includes a PhD in sociology (1996) from Carleton University. He is also the author of a book about to be released, Nasty, Brutish and Short: The Lives of Gang Members in Canada.

According to his 31-page resumé, most of Totten’s work with gangs has been in the Ottawa area and western Canada, and he says he has counselled hundreds of gang members.

Several things seem to be happening here:

1. In the most recent incident, Totten admitted he didn’t do a good job testifying. So perhaps he isn’t convincing and/or gets flustered.

2. Perhaps Totten’s knowledge is not specific enough for particular cases. While he has researched gangs, he may not know the particulars of gang activity in Toronto (or some other locations).

3. With the possibility of #1 and #2, why would either the prosecution or defense call on Totten for his expert testimony?

4. How does a judge decide whether a testifying expert has enough expertise. I’m sure there are guidelines to this but doesn’t this require the judge to assess the research ability of the expert? For example, the recent case involved questions about the methodology Totten used:

In a ruling released March 5, Clark flagged as a “problem” Totten’s data relating to the sample size of gang members he purportedly interviewed, calling it “inaccurate and misleading in several ways.”

Clark had listened for a day and a half as Misener challenged Totten about his research and methodology, including that used in the Abbey trial, and about his lack of knowledge about Toronto street gangs.

This is a very common academic argument: attack the methodology of another researcher and suggest they can’t reach the conclusions they do because the data is bad. Knowing this, many academics know they have to be able to respond to this which is why articles and books typically contain a defense of the methodology used for the study. In this case, the argument seems to be that Totten can’t really speak about Toronto gangs because there are important differences between these gangs and the ones Totten has studied. At what point is the judge convinced that Totten is not an expert for this case?

Even if the methodology is good, perhaps #1 and #2 are most important here – if the expert can’t speak well to the specific case and defend their methodology, it doesn’t matter if the expert really is an expert. Part of being an expert requires that the expert can effectively communicate their argument and the methodology behind it.

(My goal in this post is not to defend Totten or suggest his testimony should not be allowed. Rather, I was intrigued by the fact that these arguments about methodology and validity took place in court. While sociologists and researchers in other disciplines might know how the publishing system works for their own field, I assume the rules and standards in court differ even as there are some similarities between the two realms.)

Newt Gingrich: visionary professor? Historian who would be a “superior president”?

The Wall Street Journal has an interesting profile of Newt Gingrich’s days as a professor at West Georgia College. Two quick thoughts: he sounds quite ambitious as a young professor (applying to be college president during his first year) and it doesn’t sound like he followed social conventions (“brusque treatment” of his department chair, etc.).

I still find it interesting that Gingrich is using his academic credentials in his campaign:

Mr. Gingrich often says his experience as a historian would make him a superior president. During Monday’s GOP debate, he lectured “as a historian” on “a fact-based model” for revamping Social Security, citing the success of programs in Galveston, Texas, and Chile.

This could lead to some questions:

1. I thought Republicans/conservatives were more suspicious of academics who they often paint as elitist and liberal. So it isn’t actually the job or career itself that is the problem or the knowledge one has to acquire to become a professor – it is what political views the academic has?

2. What would be the best academic discipline from which to choose a President who was formerly an academic? Law seems to get a lot of attention but is this the best perspective or training to start with?

(This follow an earlier post in which I contrasted a potential presidential election between the two academics Obama and Gingrich.)

Gingrich the history professor versus Obama the law professor?

I feel something is generally being overlooked in the rise of Newt Gingrich in the polls and talk about his background. Newt is an academic who became a historian and is interested in running against a president who was once a self-described “constitutional law professor.” Let’s start with Newt’s background on Wikipedia:

Gingrich received a B.A. in history from Emory University in Atlanta in 1965, a M.A. in 1968, and a PhD in modern European history from Tulane University in New Orleans in 1971.His dissertation was entitled “Belgian Education Policy in the Congo: 1945–1960”. While at Tulane, Gingrich joined the St. Charles Avenue Baptist Church and was baptized by the Rev. G. Avery Lee.In 1970, Gingrich joined the history department at West Georgia College as an assistant professor. In 1974 he moved to the geography department and was instrumental in establishing an inter-disciplinary environmental studies program. Denied tenure, he left the college in 1978.

He has written a number of books, according to the biography at Gingrich Productions:

As an author, Newt has published twenty-three books including 13 fiction and non-fiction New York Times best-sellers.  Non-fiction books include his latest, A Nation Like No Other, in addition to Ronald Reagan: Rendezvous with DestinyTo Save America: Stopping Obama’s Secular-Socialist MachineRediscovering God in America (newly revised featuring the photography of Callista Gingrich), 5 Principles for a Successful Life, Drill Here, Drill Now, Pay Less, Real Change, A Contract with the Earth, Winning the Future: A 21st Century Contract with America, To Renew America, Lessons Learned the Hard Way, Saving Lives & Saving Money, Window of Opportunity, The Art of Transformation, and Rediscovering God in America. He is also the author of a series of historical fiction books: Gettysburg, Grant Comes East, Never Call Retreat: Lee and Grant the Final Victory, 1945, Pearl Harbor, A Novel of December the 8, Days of Infamy, To Try Men’s Souls, and his latest, Valley Forge. These novels are active history studies in the lessons of warfare based on fictional accounts of historical wartime battles and their aftermaths.

A political scientist weighs in:

He is hired as an assistant professor (a tenure track position) at West Georgia College.  While he clearly thought highly of himself (the timeline linked states that he tried to become department chair in his second year—and odd move for a variety of reasons.  These reasons include:  1)  it is difficult to be in a leadership position like that sans tenure, given that one would have to come into conflict with people who would have direct influence over tenure decisions, including senior faculty, deans, and upper administration, depending on the system in place) and, 2) new faculty have a lot of time demands, including preparing a large number of classes from scratch as well as working towards publications.

Gingrich fails to achieve tenure, meaning that his academic  career at West Georgia College was over.  Of course, from there he goes on to get elected to the House, ending his stint in academia altogether.

The interesting thing about Gingrich rather brief stint in academia is the record suggests he was never especially serious about it.  Not only did he try to become chair in his second year (an indication that he was, at a minimum, confused about how to get tenured) but he ended up running for congress during this period (a time-consuming activity).  Given the time needed to engage in teaching and scholarly output, something had to give and clearly political ambitions overtook academic ones.  Now, this is a legitimate choice for Gingrich to have made (although odd in the sense that getting the Ph.D. in the first place took a lot of work), but clearly he abandoned the academic enterprise almost at the beginning of his career (his first run for Congress was in 1974, at that point in his time at GWC that he should have been focusing intently on the fact that he would be going up for tenure and promotion soon).  As such, his claims to being a historian from a professional point of view are quite dubious.

Yes, he has published a number of books (22, I believe) but they are a  collection of co-authored novels and political/ideological tomes.  Indeed, none of the books written or co-written by Gingrich listed at Amazon would qualify as “scholarly” by actual historians…Really, he has no credibility claiming the mantle of either scholar or historian at the moment.  I can find, by the way, no evidence of any scholarly output whatsoever during his stint in the academy (I check a couple of databases that cover publications in history, but it is possible I am missing something).

A historian has similar thoughts:

But here’s what you need to know about Gingrich: He’s not a real historian. Sure, he’s got a Ph.D. in the field, and yes, Gingrich has written more than 20 books. But when he left academia for Congress in 1978, he also left behind the most basic canons of our discipline: rigor and humility. Put simply, we’re supposed to know what we’re talking about. And when we don’t, we’re supposed to say so.

That’s what I learned on my very first day of graduate school, almost a quarter-century ago. The world is infinitely complicated, a professor told us, and we’ll only be able to study a very small slice of it. And even when we think we understand that tiny piece, someone else will come along to prove us wrong.

Some of my own thoughts on this:

1. While Gingrich may not have been in academia for long, he did complete a dissertation and taught for 8 years (as far as I can tell). Both President Obama and Gingrich spent some time in academia before moving onto more success in politics. Did this background help each of them in politics?

2. I imagine many or even most historians and other academics would not support Gingrich. Since academia tends to lean away from Gingrich’s positions, I assume Gingrich would not be the favorite candidate of college professors.

2a. If this is the case, would this lead to more critical comments regarding his academic background and charges that he was just dabbling in the academy?

3. Obama and Gingrich are just two data points but could there be more academics rising to high ranks in the American political scene? How about Elizabeth Warren, Harvard law professor for over 20 years? Could a sociologist ever run for and win a higher office and how would their sociological background inform their campaign and governing strategy?

4. On the whole, is being an academic a positive thing for voters? American culture has an anti-intellectual streak as well as some negative ideas about the “educated elite.” Of course, this background might appeal to some people.

Measuring faculty productivity in sociology

A sociologist and associate dean at the University of Texas-Austin has recently put together a report on faculty productivity at his school that was undertaken to counter criticism that some faculty at the school didn’t do enough research to justify the money paid by taxpayers to support the school. The report cautions against using the same measures of productivity across disciplines:

While Musick said there was value in using the available numbers on research support, he stressed the importance of recognizing that this is valid only for some disciplines. (And one of his recommendations going forward is that the university develop better measures for research productivity of faculty members who work in disciplines without significant sources of outside funding.)

His own field of sociology is a perfect illustration of the limits of using outside funding as a measure of faculty research productivity, Musick said. Sociologists have some government support for which they can apply, but not nearly as much as do those in the physical or biological sciences, he noted. Even within fields, one’s success at obtaining funds may be based on area of expertise, not productivity. Musick said that as a medical sociologist, he has been able to win National Institutes of Health grants that some of his colleagues in sociology — people with good research agendas — could not seek.

He also said it was important to reject the idea that universities should be based only upon those fields that can attract the most outside support — even if you have a goal of producing more scientists. Musick cited as examples STEM-oriented universities such as the Massachusetts Institute of Technology and the California Institute of Technology — both of which invest significant funds in humanities and social science programs. “They recognize that universities are ecosystems,” he said. “They recognize that to produce the best scientists, they need the humanities and social sciences and the fine arts.”

It would also be helpful to keep in mind that there is even disagreement within sociology about faculty productivity. (I assume these discussions might also take place within other disciplines.) I’ve seen some heated discussion between faculty of different subfields of sociology where productivity is measured in very different ways. A book might be considered a massive achievement in one subfield while multiple journal articles are the norm in another. Plus, you could get into the quality of such publications which can also be difficult to assess. Impact factor seems to be the favored way to do this today but that has some issues and applies only to journal articles. Additionally, we could ask what the benchmark for overall productivity is: should UT-Austin match other R1 public schools and/or places like Harvard and Princeton?

Does the availability of outside funding help explain why medical sociology is a growing subfield?

Michael Bay putting together new TV show about sociology professor who studies subcultures

There are not too many sociologists in television shows or movies but producer Michael Bay is currently working on a show that has a sociologist in a prominent role:

Michael Bay is on board to exec produce an hourlong drama for the CW.

Net has bought the script “Outsiders” from writer Adam Glass, who most recently was a writer and supervising producer on “Supernatural.” He was also on staff on “Cold Case” and “The Cleaner.”

Storyline involves a quirky sociology professor with an almost savant-like expertise in subcultures. He is partnered with a young but uptight female detective and the pair solve crimes involving youths and subcultures in Los Angeles…

New CW topper Mark Pedowitz has said he’s interested in expanding the net’s reach beyond the core 18-34 female demo. A Bay series would likely bring along a male audience, and potentially tap into the international market.

Who knew that young males wanted to watch television about quirky, savant-like male sociologists? And how many odd subcultures could a show like this display?

Sociologist uses Twitter for class but are the students learning more?

Stories like this are not uncommon: professors utilizing technology to engage their students (and here is another one about clicker use).

Wendy Welch is incorporating the use of the social networking site Twitter into her cultural geography class this semester. The adjunct instructor said she decided to use the social networking site in her class after having problems with students using their phones in class for less-than-appropriate purposes.“If you can’t beat them, get ahead of them,” Welch said. “That’s the way the world works now.”…

Each student was assigned a country in Africa and asked to tweet facts about their country, such as languages and population, using designated hash tags, or categories. That way, each student only has to research one country but has access to all the information they may need from other students.

Welch also plans to have students use their mobile devices or laptops to research information during class sessions, she said…

She said she hopes to “get students to understand and participate in their own education.”

Perhaps this does increase the engagement level of the students. All professors want their students to be engaged and we don’t want to be seen as being behind the times. But, I think there is often something missing from discussions about student engagement and the use of technology in the classroom: does this actually lead to higher levels of student learning or student outcomes?

I suspect professors will always try to keep up with technology as it changes and each of these changes will be accompanied by hand-wringing. However, we need to be able to distinguish between engaging students with technology versus helping them learn. Take this Twitter example from class: do students do better on tests? Do they retain the knowledge better? Can they apply their knowledge from this particular class to other settings, particularly if the technology is not present? Does technology itself help students think more deeply about the big questions of our world?

If technology alone becomes the answer in the classroom, we will be in trouble.

How long do students keep notes from their college classes?

While discussing some of the things that he left behind in the transition between the analog and digital world, a writer includes his notes from Sociology 101:

I collected a lot of things. A large part of my identity revolved around the acquisition and accumulation of books. I also collected CDs, DVDs, comics and other cultural ephemera. I kept movie tickets, clippings of articles, flyers, interesting things I picked up. I couldn’t bear to throw these out because I thought that there might come a time when I might need something —like, say, my readings in Sociology 101 from the year 2000.

Who knew when I would have to define the sociological imagination? Or when I would need to define the political dynamics and do a comparative analysis of the authoritarian leadership styles of Lee Kuan Yew and Saddam Hussein based on my studies of Politics and Change in the Third World in 2001? Oh and there were empty liquor bottles signed by friends from the early Noughties wishing me a happy nineteenth or twentieth birthday, and lord knows a situation might arise when I might need those too.

If I was the professor of this Soc 101 class, what should be my response on hearing this? Happiness in that a former student might have turned to these notes? Depression because the student had years to look at these and never did again? Or indifference since this student seemed to collect a lot of things, not just sociology notes?

More broadly, I would be curious to know how often college students return to their books and notes from school. Does anyone have any systematic data on the subject? I suspect the data would look like a Poisson curve: most students have never returned to these sources. But couldn’t this be a measure of the “effectiveness” or “success” of a particular class, an outcome that colleges and professors might be interested in knowing about? Typically, we get information on evaluations forms from the closing moments of class, a time when students might be able to judge the immediate effect of a class but can shed little light on the longer-lasting impact of a particular course. Imagine if we found that a more popular sociological text like Gang Leader For a Day was popular in the short-term but a text like The Truly Disadvantaged stuck with students for years. Both outcomes could be desirable – a short-term book or lecture can draw people into the subject or enhance the classroom experience while a longer-term book or lecture can influence lives down the road – but are qualitatively different pieces of information.

Perhaps this could all be explained by personality types: there are people who keep things from the past and those who do not. But I suspect that professors would like to think that they have the potential in many lectures or in the sources they put in front of students to influence any student for years.

Syllabus bloat: the ever-lengthening college syllabus

A professor discusses the reasons why college syllabi keep getting bigger:

Nowadays my course syllabi tend to run to many pages and always include a punctilious day-by-day calendar of the semester stipulating, for example, precisely which pages in what book students need to have read for class.  My instructions to students concerning formal written work have also become replete with prescription in a way that I would not have thought necessary even ten years ago.  Colleagues concur that instructors at the state-college level can take little or nothing for granted about student preparedness and that everything, absolutely everything, must be spelled out in advance.  Without abundant guidance and prescription, students complain of being lost, as perhaps they are, or of “not understanding what the professor wants,” as is perhaps the case…

First-year college students have a drastically diminished vision of what higher education portends for them.  The idea of discipline that enabled my UCLA instructors to assume procedural competency in their students, and that enabled most students to acquit themselves during the term with only a minimal syllabus, no longer exists…

The enlargement of the syllabus also stems from the need to define, explain, and insofar as possible justify the course itself, something that no syllabus from my undergraduate career ever bothered to do.  The syllabus of my survey of ancient literature (“Western Heritage”) addresses the basic notion of historical indebtedness, the idea of continuity of insight, and of the dignity of knowledge as opposed to ignominy of ignorance.  The syllabus also addresses the difficulty of reading; it tells students that an epic poem by Homer or a philosophical dialogue by Plato is not like a TV drama or a movie, in which in the first few minutes, one can predict the remainder…

Some of this effort—and much of the hypertrophied syllabus—is precautionary. It is precautionary on behalf of students, who, from day one, will know in advance every requirement and assignment of the course. It is also precautionary on behalf of the syllabus-writer, who seeks protection from petulant students claiming they never knew the schedule or failed to receive procedural knowledge concerning the semester.  Syllabus in hand, no one can plead ignorance.

The general idea is this: today’s college students need college explained to them, point by point. This could quickly turn into a generational argument that is bigger than just college classes: the role of college has changed from a place of learning to four years of job training. Society, and consequentially, college students simply don’t know what college is about when they should and professors have to do the extra work to explain it. This could also be tied to the issue that a number of college students are not ready to do college-level work.

There may be some truth to this but, as the article hints at, there could be good reasons to have longer syllabi:

1. Expectations are made clear from the beginning. This could cut down student’s anxiety as there is less “guesswork” involved. If a relatively short document (compared to books/journal articles) can help eliminate ignorance, why not?

2. Why not have a short part of the syllabus that explains what the class is about? Certain subjects, like sociology, are relatively unknown and a one or two paragraph introduction can give students a engaging foundation.

3. I like having the day-to-day calendar for myself so why not provide it for the students as well? Perhaps this is just because I like to be organized.

4. I wonder if a detailed and longer syllabi just by its thoroughness conveys to students the importance of the task. Some students may groan at seeing how much there is to read but others will feel the gravity.

We could transfer these ideas to another context: would many employees find it acceptable if they came to work each day with little idea of what to expect? On one hand, we should promote internal motivation but some structure is helpful. We can rue the loss of “gravity” and “mystery” that students have or feel regarding college or we can try to convey these ideas in our syllabi and what we say and do in the classroom.

h/t Instapundit

Argument: college students are poor writers because they ape the academic prose of their professors

An English professor argues that the problem with what students learn about writing in college is that they learn to write in the style of professors:

This Wall Street Journal article implies that our poor communication skills fresh out of college result from simple laziness or stupidity on our part. But Dr. Richard Lanham, professor emeritus of English, UCLA, believes the problem is not lack of learning.

Rather, he believes, when we are undergraduates, we learn all too well: we learn to ape the bureaucratic, academic, clear-as-swamp-water prose of our professors. He writes in his book, Revising Prose (in my opinion the single best practical book on prose writing in existence):

“Much bad writing today comes not from the conventional sources of verbal dereliction—sloth, original sin, or native absence of mind—but from stylistic imitation. It is learned, an act of stylistic piety, which imitates a single style, the bureaucratic style I have called The Official Style. . . .”

And to drive home his point, Lanham cites a passage from Talcott Parsons. This particular passage appears to be from Parsons’ book The Social System and you can see the passage here on page 7 in Google Book. (Interestingly, this passage is edited. Yes, it is difficult but why not cite a complete passage rather than starting at the end of a paragraph and cutting some of another?) Lanham says this about Parsons’ passage:

This is not prose. This is the systematic abuse of prose. Anyone hoping to learn writing should stay a thousand miles away from people who write in such a manner. That is, they should stay a thousand miles away from most university professors.

The author of this article goes on to cite Sokol’s Social Text hoax, argue that computers could teach academic writing (though it may not make any sense), and suggest that college students need to get beyond typical college classes.

The trend these days in college seems to be to train students to write in discipline specific styles: if you are a biology major, write a biologist, a sociology major, write like a sociologist (though not like Parsons), and son. This makes some sense for faculty as our professional careers are based on particular writing styles. It may not make sense for all students as they may not be interested in graduate school or professional careers. But, if students are specific majors, part of their senior-level competency should be an ability to communicate within that sphere.

However, the problem referenced at the beginning at the article may not be the same problem cited in the rest of the article. The article begins by citing a Wall Street Journal article saying that businesses are having a hard time finding possible employees who can write and speak effectively. But this problem is not the problem of academic writing; rather, it is an issue of basic communication. Employers aren’t saying that college grads are writing postmodern gibberish; rather, they are saying that applicants write as though they are interacting with friends. Academic writing may be problematic (and it is recurring issue within sociology though most work doesn’t read like Parsons) but the problem employers are discussing is a few steps before academic writing.

If college students are trained in any discipline specific style and can successfully write within that world, that’s better than not being able to write any prose. Theoretically, students should be able to write basic prose before they write discipline specific prose but some prose is better than no prose, right? Students need to learn the basics, thesis, supporting evidence, etc., and perhaps the average professor is not able to offer much help in this since they are immersed in academic language.

For students to move beyond simply aping the academic style of their professors, they need to practice writing a lot. Even while there might be guidelines and norms within particular disciplines, there is still freedom within these areas to exhibit some personal style. Within sociology, for example, there is a wide range between ethnographic and statistical/mathematical work. Styles are not formulas. Of course, more writing means more work for students and professors. But there is evidence that an increase in writing within academic courses leads to better educational outcomes.

On the whole, I think this is a bigger issue than academics passing along bad prose to their students. Sociologists could indeed do better than writing like Talcott Parsons. All classes should help students write basic, effective prose that can be used outside of the college classroom. How all this could and should be worked out within a typical four-year degree needs to be developed.