Illinois lost residents 2010 to 2020; discrepancies in year to year estimates and decennial count

Illinois lost residents over the last decade. But, different Census estimates at different times created slightly different stories:

Photo by Nachelle Nocom on Pexels.com

Those estimates showed Illinois experiencing a net loss of 9,972 residents between 2013 and 2014; 22,194 residents between 2014 and 2015; 37,508 residents between 2015 and 2016; about 33,700 residents between 2016 and 2017; 45,116 between 2017 and 2018; 51,250 between 2018 and 2019; and 79,487 between 2019 and 2020…

On April 26, the U.S. Census Bureau released its state-by-state population numbers based on last year’s census. These are the numbers that determine congressional apportionment. Those numbers, released every 10 years, show a different picture for Illinois: a loss of about 18,000 residents since 2010.

What’s the deal? For starters, the two counting methods for estimated annual population and the 10-year census for apportionment are separate. Apples and oranges. Resident population numbers and apportionment population numbers are arrived at differently, with one set counting Illinois families who live overseas, including in the military, and one not.

Additionally, the every-10-years number is gathered not from those county-by-county metrics but from the census forms we fill out and from door-to-door contacts made by census workers on the ground.

The overall story is the same but this is a good reminder of how different methods can produce different results. Here are several key factors to keep in mind:

  1. The time period is different. One estimate comes every year, one comes every ten years. The yearly estimates are helpful because people like data. That does not necessarily mean the yearly estimates can be trusted as much as the other ones.
  2. The method in each version – yearly versus every ten years – is different. The decennial data involves more responses and requires more effort.
  3. The confidence in the two different kinds of estimates is different because of #2. The ten year estimates are more valid because they collect more data.

Theoretically, the year-to-year estimates could lead to a different story compared to the decennial estimates. Imagine year-to-year data that told of a slight increase in population while the ten-year numbers provided a slight decrease in population. This does not mean the process went wrong there or in the narrative where the yearly and ten-year estimates agreed. With estimates, researchers are trying their best to measure the full population patterns. But, there is some room for error.

That said, now that Illinois is known as one of the three states that lost population over the last decade, it will be interesting to see how politicians and business leaders respond. I can predict some of the responses already as different groups have practiced their talking points for years. Yet, the same old rhetoric may not be enough as these figures paint Illinois in a bad light when population growth is good in the United States.

Researchers adjust as Americans say they are more religious when asked via phone versus responding online

Research findings suggest Americans answer questions about religiosity differently depending on the mode of the survey:

Photo by mentatdgt on Pexels.com

Researchers found the cause of the “noise” when they compared the cellphone results with the results of their online survey: social desirability bias. According to studies of polling methods, people answer questions differently when they’re speaking to another human. It turns out that sometimes people overstate their Bible reading if they suspect the people on the other end of the call will think more highly of them if they engaged the Scriptures more. Sometimes, they overstate it a lot…

Smith said that when Pew first launched the trend panel in 2014, there was no major difference between answers about religion online and over the telephone. But over time, he saw a growing split. Even when questions were worded exactly the same online and on the phone, Americans answered differently on the phone. When speaking to a human being, for example, they were much more likely to say they were religious. Online, more people were more comfortable saying they didn’t go to any kind of religious service or listing their religious affiliation as “none.”…

After re-weighting the online data set with better information about the American population from its National Public Opinion Reference Survey, Pew has decided to stop phone polling and rely completely on the online panels…

Pew’s analysis finds that, today, about 10 percent of Americans will say they go to church regularly if asked by a human but will say that they don’t if asked online. Social scientists and pollsters cannot say for sure whether that social desirability bias has increased, decreased, or stayed the same since Gallup first started asking religious questions 86 years ago.

This shift regarding studying religion highlights broader considerations about methodology that are always helpful to keep in mind:

  1. Both methods and people/social conditions change. More and more surveying (and other data collection) is done via the Internet and other technologies. This might change who responds, how people respond, and more. At the same time, actual religiosity changes and social scientists try to keep up. This is a dynamic process that should be expected to change over time to help researchers get better and better data.
  2. Social desirability bias is not the same as people lying to researchers or being dishonest with researchers. That implies an intentional false answer. This is more about context: the mode of the survey – phone or online – influences who the respondent is responding to. And with a human interaction, we might respond differently. In an interaction, we with impression management in mind where we want to be viewed in particular ways by the person with whom we are interacting.
  3. Studying any aspect of religiosity benefits from multiple methods and multiple approaches to the same phenomena under study. A single measure of church attendance can tell us something but getting multiple data points with multiple methods can help provide a more complete picture. Surveys have particular strengths but they are not great in other areas. Results from surveys should be put alongside other data drawn from interviews, ethnographies, focus groups, historical analysis, and more to see what consensus can be reached. All of this might be out of the reach of individual researchers or single research projects but the field as a whole can help find the broader patterns.

The Census as national process yet works better with local census takers

Among other interesting tidbits about how data was collected for the 2020 census, here is why it is helpful for census takers to be from the community in which they collect data:

Photo by Sunyu Kim on Pexels.com

As it turns out, the mass mobilization of out-of-state enumerators is not just uncommon, but generally seen as a violation of the spirit of the census. “One of the foundational concepts of a successful door-knocking operation is that census takers will be knowledgeable about the community in which they’re working,” Lowenthal explained. “This is both so they can do a good job, because they’ll have to understand local culture and hopefully the language, but also so that the people who have to open their doors and talk to them have some confidence in them.”

Going door to door is a difficult task. Some connection to the community could help convince people to cooperate. And when cooperation equals higher response rates and more accurate data, local knowledge is good.

As the piece goes on to note, this does not mean that outside census takers could not help. Having more people going to every address could help boost response rates even if the census takers were from a different part of the country.

I wonder how much local knowledge influences the response rates from proxies, other people who can provide basic demographic information when people at the address do not respond:

According to Terri Ann Lowenthal, a former staff director for the House census oversight subcommittee, 22 percent of cases completed by census takers in 2010 were done so using data taken from proxies. And of those cases, roughly a quarter were deemed useless by the Census Bureau. As a result, millions of people get missed while others get counted twice. These inaccuracies tend to be more frequent in urban centers and tribal areas, but also, as I eventually learned, in rural sections of the country.

It is one thing to have the imprimatur of the Census when talking with a proxy; it would seem to be a bonus to also be a local.

More broadly, this is a reminder of how an important data collection process depends in part on local workers. With a little bit of inside knowledge and awareness, the Census can get better data and then that information can effectively serve many.

Combating abysmally low response rates for political polling

One pollster describes the difficulty today in reaching potential voters:

Photo by Breakingpic on Pexels.com

As the years drifted by, it took more and more voters per cluster for us to get a single voter to agree to an interview. Between 1984 and 1989, when caller ID was rolled out, more voters began to ignore our calls. The advent of answering machines and then voicemail further reduced responses. Voters screen their calls more aggressively, so cooperation with pollsters has steadily declined year-by-year. Whereas once I could extract one complete interview from five voters, it can now take calls to as many as 100 voters to complete a single interview, even more in some segments of the electorate…

I offer my own experience from Florida in the 2020 election to illustrate the problem. I conducted tracking polls in the weeks leading up to the presidential election. To complete 1,510 interviews over several weeks, we had to call 136,688 voters. In hard-to-interview Florida, only 1 in 90-odd voters would speak with our interviewers. Most calls to voters went unanswered or rolled over to answering machines or voicemail, never to be interviewed despite multiple attempts.

The final wave of polling, conducted Oct. 25-27 to complete 500 interviews, was the worst for cooperation. We could finish interviews with only four-tenths of one percent from our pool of potential respondents. As a result, this supposed “random sample survey” seemingly yielded, as did most all Florida polls, lower support for President Trump than he earned on Election Day.

After the election, I noted wide variations in completion rates across different categories of voters, but nearly all were still too low for any actual randomness to be assumed or implied.

This is a basic Research Methods class issue: if you cannot collect a good sample, you are going to have a hard time reflecting reality for the population.

Here is the part I understand less. This is not a new issue. As noted above, response rates have been falling for decades. Part of it is new technology. Some of it involves new behavior, such as ignoring phone calls or distrust of political polling. The amount of polling and data collection that takes place now can lead to survey fatigue.

But, it is interesting that the techniques used to collect this data are roughly the same. Of course, it has moved from land lines to cell phones and perhaps even texting or recruited online pools of potential voters. The technology has changed some but the idea is similar in trying to reach out to a broad set of people and hope a representative enough sample responds.

Perhaps it is time for new techniques. The old ones have some advantages including the ability to relatively quickly reach a large number of people and researchers and consultants are used to these techniques. And I do not have the answers for what might work better. Researchers embedded in different communities who could collect data over time? Finding public spaces frequented by diverse populations and approaching people there? Working more closely with bellwhether or representative places or populations to track what is going on there?

Even with these low response rates, polling can still tell us something. It is not as bad as picking randomly or flipping a coin. Yet, it is not accurate enough in recent years. If researchers want to collect valid and reliable polling data in the future, new approaches may be in order.

Fewer plane flights, worse weather forecasts, collecting data

The consequences of COVID-19 continue: with fewer commercial airline flights, weather models have less data.

white clouds and blue sky

Photo by Ithalu Dominguez on Pexels.com

During their time in the skies, commercial airplanes regularly log a variety of meteorological data, including air temperature, relative humidity, air pressure and wind direction — data that is used to populate weather prediction models…

With less spring meteorological data to work with, forecasting models have produced less accurate predictions, researchers said. Long-term term forecasts suffered the most from a lack of meteorological data, according to the latest analysis…

Forecast accuracy suffered the most across the United States, southeast China and Australia, as well as more remote regions like the Sahara Desert, Greenland and Antarctica.

Though Western Europe experienced an 80 to 90 percent drop in flight traffic during the height of the pandemic, weather forecasts in the region remained relatively accurate. Chen suspects the region’s densely-packed network of ground-based weather stations helped forecasters continue to populate models with sufficient amounts of meteorological data.

Models, whether for pandemics or weather, need good input. Better data up front helps researchers adjust models to fit past patterns and predict future outcomes. Absent of data, it can be hard to fit models, especially in complex systems like weather.

As noted above, there are other ways to obtain weather data. Airplanes offered a convenient way to collect data: thousands of regular flights could lead to a lot of data. In contrast, constructing ground stations would require more resources in the short-term.

Yet, any data collector needs to remain flexible. One source of data can disappear, leading to a new approach. Or, a new opportunity might arise and switching methods makes sense. Or, those studying and predicting weather could develop multiple good sources of data that could options or redundancy amid black swan events.

Few may recognize all of this is happening. Weather forecasts will continue. Behind the scenes, we might even get better weather models in the long run as researchers and meteorologists adjust.

Font sizes, randomly ordered names, and an uncertain Iowa poll

Ahead of the Iowa caucuses yesterday, the Des Moines Register had to cancel a final poll just ahead of the voting due to problems with administering the survey:

Sources told several news outlets that they figured out the whole problem was due to an issue with font size. Specifically, one operator working at the call center used for the poll enlarged the font size on their computer screen of the script that included candidates’ names and it appears Buttigieg’s name was cut out from the list of options. After every call the list of candidates’ names is reordered randomly so it isn’t clear whether other candidates may have been affected as well but the organizers were not able to figure out whether it was an isolated incident. “We are unable to know how many times this might have happened, because we don’t know how long that monitor was in that setting,” a source told Politico. “Because we do not know for certain—and may not ever be able to know for certain—we don’t have confidence to release the poll.”…

In their official statements announcing the decision to nix the poll, the organizers did not mention the font issue, focusing instead on the need to maintain the integrity of the survey. “Today, a respondent raised an issue with the way the survey was administered, which could have compromised the results of the poll. It appears a candidate’s name was omitted in at least one interview in which the respondent was asked to name their preferred candidate,” Register executive editor Carol Hunter said in a statement. “While this appears to be isolated to one surveyor, we cannot confirm that with certainty. Therefore, the partners made the difficult decision to not to move forward with releasing the Iowa Poll.” CNN also issued a statement saying that the decision was made as part of their “aim to uphold the highest standards of survey research.”

This provides some insight into how these polls are conducted. The process can include call centers, randomly ordered names, and a system in place so that the administrators of the poll can feel confident in the results (even as there is always a margin of error). If there is a problem in the system, the opinions of those polled may not match what the data says. Will the future processes not allow individual callers to change the font size?

More broadly, a move like this could provide more transparency and ultimately trust regarding political polling. The industry faces a number of challenges. Would revealing this particular issue cause people to wonder how often this happens or reassure them that pollsters are concerned about good data?

At the same time, it appears that the unreported numbers still had an influence:

Indeed, the numbers widely circulating aren’t that different from last month’s edition of the same poll, or some other recent polls. But to other people, both journalists and operatives, milling around the lobby of the Des Moines Marriott Sunday night, the impact had been obvious.

Here are what some reporters told me about how the poll affected their work:

• One reporter for a major newspaper told me they inserted a few paragraphs into a story to anticipate results predicted by the poll.

• A reporter for another major national outlet said they covered an Elizabeth Warren event in part because she looked strong in the secret poll.

• Another outlet had been trying to figure out whether Amy Klobuchar was surging; the poll, which looked similar to other recent polling, steered coverage away from that conclusion.

• “You can’t help it affecting how you’re thinking,” said another reporter.

asdf

Surprise! The best suburbs in America are wealthy, educated, and in regions with reasonable costs of living

The Niche 2019 Best Places to Live falls into some of the same patterns of similar lists of highlighting already well-off communities with a high quality of life. Part of the reason is the methodology:

Niche2019BestPlacestoLive

If this is what Niche and Money and other want to look for in terms of data and how it is weighted, they are going to consistently churn out lists of similar kinds of communities. The “best” suburbs and small towns in certain regions, those with higher housing prices, will find it hard to make the list. A certain amount of diversity is acceptable but not too much and it is related to social class. In other words, these are lists that might be intended for middle to upper-class suburbanites who are looking for safe, quiet, and enriching places to live.

So, perhaps instead of calling these the “Best Places to Live,” how about: “Aspirational Places for Middle- to Upper-Class Families?” Or, how about more lists that address hidden gems, communities that wouldn’t make a list like this due to one factor or another but are still great places? Or, how about ones that weight certain factors a lot higher, like “The Best Diverse Suburbs” or “The Best Suburbs for Housing Opportunities.”

Ultimately, these lists tend to reinforce cultural narratives about the places in which Americans most want to live and where the American Dream can be found. No doubt these magazines and sites need to sell copy – there are Americans who want to move to these top suburbs. But, there are also hundreds of other great places to live in the United States that do not always fit the longstanding suburban mold of mostly white, wealthy, educated, and quiet.

Teaching how science and research actually works

As a regular instructor of Statistics and Social Research classes, I took note at this paragraph in a recent profile of Bruno Latour:

Latour believes that if scientists were transparent about how science really functions — as a process in which people, politics, institutions, peer review and so forth all play their parts — they would be in a stronger position to convince people of their claims. Climatologists, he says, must recognize that, as nature’s designated representatives, they have always been political actors, and that they are now combatants in a war whose outcome will have planetary ramifications. We would be in a much better situation, he has told scientists, if they stopped pretending that “the others” — the climate-change deniers — “are the ones engaged in politics and that you are engaged ‘only in science.’ ” In certain respects, new efforts like the March for Science, which has sought to underscore the indispensable role that science plays (or ought to play) in policy decisions, and groups like 314 Action, which are supporting the campaigns of scientists and engineers running for public office, represent an important if belated acknowledgment from today’s scientists that they need, as one of the March’s slogans put it, to step out of the lab and into the streets. (To this Latour might add that the lab has never been truly separate from the streets; that it seems to be is merely a result of scientific culture’s attempt to pass itself off as above the fray.)

Textbooks on Statistics and Social Research say there are right ways and wrong ways to do the work. There are steps to follow, guidelines to adhere to, clear cut answers on how to do the work right. It is all presented in a logical and consistent format.

There are hints that this may not happen all the time. Certain known factors as well as unknown issues can push a researcher off track a bit. But, to do a good job, to do work that is scientifically interesting and acceptable to the scientific community, you would want to stick to the guidelines as much as possible.

This provides a Weberian ideal type of how science should operate. Or, perhaps the opposite ideal type occasionally provides a contrast. The researcher who committed outright fraud. The scholar who stepped way over ethical boundaries.

I see one of my jobs of teaching these classes as providing how these steps work out in actuality. You want to follow those guidelines but here is what can often happen. I regularly talk about the constraints of time and money: researchers often want to answer big questions with ideal data and that does not always happen. You make mistakes, such as in collecting data or analyzing results. You send the manuscript off for review and people offer all sorts of suggestions of how to fix it. The focus of the project and the hypothesis changes, perhaps even multiple times. It takes years to see everything through to publication.

On one hand, students often want the black and white presentation because it offers clear guidelines. If this happens, do this. On the other hand, presenting the cleaner version is an incomplete education into how research works. Students need to know how to respond when the process does not go as planned and know that this does not necessarily mean their work is doomed.

Scientific research is not easy nor is it always clear cut. Coming back to the ideal type concept, perhaps we should present it as we aspire to certain standards and particular matters may be non-negotiable but there are parts of the process, sometimes small and sometimes large, that are more flexible depending on circumstances.

Speculating on why sociology is less relevant to the media and public than economics

In calling for more sociological insight into economics, a journalist who attended the recent ASA meetings in Philadelphia provides two reasons why sociology lags behind economics in public attention:

Economists, you see, put draft versions of their papers online seemingly as soon as they’ve finished typing. Attend their big annual meeting, as I have several times, and virtually every paper discussed is available beforehand for download and perusal. In fact, they’re available even if you don’t go to the meeting. I wrote a column two years ago arguing that this openness had given economists a big leg up over the other social sciences in media attention and political influence, and noting that a few sociologists agreed and were trying to nudge their discipline — which disseminates its research mainly through paywalled academic journals and university-press books — in that direction with a new open repository for papers called SocArxiv. Now that I’ve experienced the ASA annual meeting for the first time, I can report that (1) things haven’t progressed much since 2016, and (2) I have a bit more sympathy for sociologists’ reticence to act like economists, although I continue to think it’s holding them back.

SocArxiv’s collection of open-access papers is growing steadily if not spectacularly, and Sociological Science, an open-access journal founded in 2014, is carving out a respected role as, among other things, a place to quickly publish articles of public interest. “Unions and Nonunion Pay in the United States, 1977-2015” by Patrick Denice of the University of Western Ontario and Jake Rosenfeld of Washington University in St. Louis, for example, was submitted June 12, accepted July 10 and published on Wednesday, the day after it was presented at the ASA meeting. These dissemination tools are used by only a small minority of sociologists, though, and the most sparsely attended session I attended in three-plus days at their annual meeting was the one on “Open Scholarship in Sociology” organized by the University of Maryland’s Philip Cohen, the founder of SocArxiv and one of the discipline’s most prominent social-media voices. This despite the fact that it was great, featuring compelling presentations by Cohen, Sociological Review deputy editor Kim Weeden of Cornell University and higher-education expert Elizabeth Popp Berman of the State University of New York at Albany, and free SocArxiv pens for all.

As I made the rounds of other sessions, I did come to a better understanding of why sociologists might be more reticent than economists to put their drafts online. The ASA welcomes journalists to its annual meeting and says they can attend all sessions where research is presented, but few reporters show up and it’s clear that most of those presenting research don’t consider themselves to be speaking in public. The most dramatic example of this in Philadelphia came about halfway through a presentation involving a particular corporation. The speaker paused, then asked the 50-plus people in the room not to mention the name of said corporation to anybody because she was about to return to an undercover job there. That was a bit ridiculous, given that there were sociologists live-tweeting some of the sessions. But there was something charming and probably healthy about the willingness of the sociologists at the ASA meeting to discuss still-far-from-complete work with their peers. When a paper is presented at an economics conference, many of the discussant’s comments and audience questions are attempts to poke holes in the reasoning or methodology. At the ASA meeting, it was usually, “This is great. Have you thought about adding …?” Also charming and probably healthy was the high number of graduate students presenting research alongside the professors, which you don’t see so much at the economists’ equivalent gathering.

All in all — and I’m sure there are sociological terms to describe this, but I’m not familiar with them — sociology seems more focused on internal cohesion than economics is. This may be partly because it’s what Popp Berman calls a “low-consensus discipline,” with lots of different methodological approaches and greatly varying standards of quality and rigor. Economists can be mean to each other in public yet still present a semi-united face to the world because they use a widely shared set of tools to arrive at answers. Sociologists may feel that they don’t have that luxury.

Disciplinary differences can be mystifying at times.

I wonder about a third possible difference in addition to the two provided: different conceptions in sociology and economics about what constitutes good arguments and data (hinted at above with the idea of “lots of different methodological approaches and greatly varying standards of quality and rigor.”) Both disciplines do aspire to the idea of social science where empirical data is used to test hypotheses about human behavior, usually in collectives, works. But, this is tricky to do as there are numerous pitfalls along the way. For example, accurate measurement is difficult even when a researcher has clearly identified a concept. Additionally, it is my sense that sociologists as a whole may be more open to qualitative and quantitative data (even with occasional flare-ups between researchers studying the same topic yet falling in different methodological camps). With these methodological questions, sociologists may feel they need more time to connect their methods to a convincing causal and scientific argument

A fourth possible reason behind the differences (also hinted at above with the idea of economists having a “semi-united face” to present): sociology has a reputation as a more left-leaning discipline. Some researchers may prefer to have all their ducks in a row before they expose their work to full public scrutiny. The work of economists is more generally accepted by the public and some leaders while sociology regularly has to work against some backlash. (As an example, see conservative leaders complain about sociology excusing poor behavior when the job of the discipline is to explain human behavior.) Why expose your work to a less welcoming public earlier when you could take a little more time to polish the argument?

Online survey panels in first-world countries versus developing nations

While reading about the opposition Canadians have to self-driving cars, I ran into this explanation from Ipsos about conducting online surveys in countries around the world:

IpsosOnlineSurveybyCountry2019

Having online panels is a regular practice among survey organizations. However, I do not recall seeing an explanation like this regarding differences in online panels across countries. The online sample in non-industrialized countries is simply unrepresentative as it reflects “a more ‘connected’ population.” Put another way, the online panel in places like Brazil, China, Russia, and Saudi Arabia reflects the upper class and people who live more like Westerners and not the vast majority of their population. Then, the sample is also smaller in these countries: 500+ rather than 1000+. Finally, it would be interesting to see how much the data needs to be weighted to “best reflect the demographic proile of the adult population.”

With all these caveats, is an online panel in a non-industrialized country worth it?