Statistical illiteracy in film studies
UPDATE: The paper at the end of this post is now available for advance access at Literary and Linguistic Computing, and can be cited as: The log-normal distribution is not an appropriate parametric model for shot length distributions of Hollywood films, Literary and Linguistic Computing, Advance Access published December 13, 2012, doi:10.1093/llc/fqs066. I will put up the paginated reference when the print version is released.
This week’s post combines two very different approaches to film studies: on the one hand we have outright anger, and then we have proper research. Both are equally important.
Are you au fait with this?
I wrote a second version of my paper examining the impact of sound technology on shot length distributions of Hollywood films using a larger sample of films. I also expanded on the methodology used (Mann-Whitney U, probability of superiority, etc.) since this has been highlighted as a problem before. (The original version is here). Having finished the article I sent it to The New Soundtrack at Edinburgh University Press. The article was turned down 24 hours later, and the reason given for rejecting the article was that, in the editors opinion,
our readership might not be quite au fait with the methodology you describe in the piece.
Nothing about the quality of the piece; just the lack of confidence the editors have in their readership.
What sort of intellectual cowardice is this? Are film scholars afraid of learning new things? Or is it that journal editors have such a low opinion of their readership that they need to protect them from anything that might be new or unusual ? Does the readership of The New Soundtrack really not know what a ‘median’ is? Is there no sense of intellectual discovery?
If I was part of the readership of The New Soundtrack I would be very unhappy with this. Presumably, if I am a subscriber to an academic journal I am (or at least consider myself to be) a reasonably intelligent person capable of thinking and learning for myself. (Perhaps I am part of the sophisticated readership of Screen as well). Do I really need someone to decide for me what I might or might not be au fait with? Now I’m wondering what other research I’ve missed out on because an editor has decided what I might or might not be comfortable with.
Have you ever heard anything so pathetic? I have, and this is now the third time I have had a journal reject one of my articles because of the use of statistical methods (see here).
Statistical literacy is defined as
the ability to understand and critically evaluate statistical results that permeate our daily lives – coupled with the ability to appreciate the contributions that statistical thinking can make in public and private, professional and personal decisions (Wallman 1993: 1).
This is relevant to film studies because we encounter statistical information in diverse contexts. Statistics is relevant in film and television studies in the study of film style, in researching the economics of the film industry, in audience studies, and in scientific research on cognition and perception in the cinema. Understanding a great deal of research of film studies assumes that you have at least some degree statistical literacy.
Gal (2002) argues that statistical literacy comprises two elements:
- a knowledge component, in which individuals have the ability to ‘interpret and critically evaluate statistical information, data-related arguments, or stochastic phenomena which they may encounter in diverse contexts’ (2); and
- a dispositional component, in which individuals develop a questioning attitude to research that purports to be based on data, a positive view of themselves as ‘individuals capable of statistical and probabilistic reasoning as well as a willingness and interest to “think statistically” in relevant situations’ and ‘a belief in the legitimacy of critical action even if they have not learnt much formal statistics or mathematics’ (19).
Arguably the dispositional component is the most important since the willingness to the think statistically is a pre-requisite for learning statistical concepts.
It is clear that the editors of The New Soundtrack have concerns about the statistical literacy of their readership. The editors apparently assume their readership will not have the required statistical knowledge to understand research presenting statistical analysis of data, and – much more damaging – they do not believe their readership has the capability or willingness to think statistically.
Altman (2002) notes readers assume that articles published in peer-review journals are scientifically sound. But in order to make an informed interpretation of the material that appears in peer-reviewed sources we need to be able to intelligently interpret it. This means that statistical literacy is a must for film studies, and it is a topic we will return to repeatedly over the rest of the year. In the next section I demonstrate how knowledge of statistical concepts and process and a questioning attitude are essential in judging the importance of research in film studies.
The lognormal dragon is slain (again)
An example of the importance of developing statistical literacy in film studies comes in the form of a new book to be published this year featuring a chapter by Jordan De Long, Kaitlin L. Brunick, and James E. Cutting. The link to an online version of this paper is below Figure 1. I won’t explain the statistical concepts in detail, but I have provided links for statistical terms and concepts. I will assume you are an intelligent reader capable of and willing to learn for yourself.
In their chapter on film style, the authors make the following statement about the average shot length (ASL) as a statistic of film style:
Despite being the popular metric, ASL may be inappropriate because the distribution of shot lengths isn’t a normal bell curve, but rather a highly skewed, lognormal distribution. This means that while most shots are short, a small number of remarkably long shots inflate the mean. This means that the large majority of shots in a film are actually below average, leading to systematic over-estimation of individual film’s shot length. A better estimate is a film’s Median Shot Length, a metric that … provides a better estimate of shot length.
In support of this statement they include a graph that purports to show how the shot length distribution of one film is lognormal. This is the only piece of evidence they provide.
Figure 1 Histogram of shot lengths in A Night at the Opera with a fitted lognormal distribution from De Long J, Brunick KL, and Cutting JE 2012 Film through the human visual system: finding patterns and limits, in JC Kaufman and DK Simonton (eds.) The Social Science of Cinema. New York: Oxford University Press: in press. This graph was downloaded from the online version of this paper available at http://people.psych.cornell.edu/~jec7/pubs/socialsciencecinema.pdf.
Clearly, the authors have assumed their readership has a fairly sophisticated level of statistical literacy. They present their argument assuming you will be able to understand it or be capable of learning the relevant concepts. An entirely reasonable way in which to present an argument in a research output, and presumably an attitude that comes from their scientific (rather than film studies) background.
It’s just a shame it’s not true.
The key fact to bear in mind is that a variable (such as shot length) is said to be lognormally distributed if its logarithm is normally distributed, as this allows us to apply a logarithmic transformation and then to try to determine if it is normally distributed.
Figure 2 Exploratory data analysis of shot lengths in A Night at the Opera
In the top left panel we see the histogram of the log-transformed shot lengths and it is immediately obvious this data set is not normally distributed. If De Long, Brunick, and Cutting are right, then this chart should be symmetrical about the mean. The histogram remains skewed even after the transformation is applied. The same pattern can be seen from the kernel density estimate (top right), which is clearly not symmetrical.
The normal probability plot (bottom left) shows the same pattern. If the data does come from a lognormal distribution then the points in this plot will be a straight line. In fact, they will lie along the red line shown in the plot. It is obvious that this is not the case and that the data points show clear evidence of a skewed data set. In the lower tail of the fitted lognormal distribution underestimates the number of shorter takes, while the upper tail overestimates the number of longer takes. Definitively NOT lognormal.
Finally, the box plot (bottom right) clearly shows the distribution is asymmetrical with outliers in the upper tail of the distribution. This is a good example of the fat that log-transforming does not always remove the skew from a data set of deal with the problem of outliers.
The marks below histogram, kernel density, and box plot (called a rug) indicate the actual values of the log-transformed shot lengths.
We can also apply formal statistical tests of the hypothesis that the shot length distribution is lognormal. Because a variable is lognormally distributed if its logarithm is normally distributed, then all we have to do is to apply normality tests to the transformed data.
The Shapiro-Francia test is based on the squared correlation of the theoretical and sample quantiles in the probability plot in Figure 2. For this film, the test statistic is 0.9585 and p = <0.01, so it is extremely unlikely that this data comes from a lognormal distribution and we have sufficient evidence to reject this hypothesis.
The Jarque-Bera test does the same thing in a different way. This test looks at the skew (its symmetry) and the kurtosis (the shape of its peak) of the data. For A Night at the Opera, the result of this test is 62.48 (p = <0.01) and again we have sufficient evidence to reject the hypothesis that this data comes from a lognormal distribution.
In summary, De Long, Brunick, and Cutting present a single piece of evidence in support of the assertion that shot length distributions are lognormal and its wrong. In fact, if you wanted to write a book about how shot length distributions are not lognormal and wanted to put an example of this on the cover then A Night at the Opera would be the film you would use.
Clearly, there is a problem with the histogram in Figure 1 that shows the shot length data on an untransformed scale. The reason for applying a logarithmic transformation is to make it easier to see the structure of the data, so why not view it on a logarithmic scale? When we view the data on a logarithmic scale we come to the opposite conclusion than that presented above. It requires statistical literacy on the part of the reader to question if this is an appropriate way of presenting data and to question the interpretation presented by the authors.
Obviously we cannot say that just because we can show that one film is not lognormally distributed this is true for all films. In order to properly assess the validity of De Long, Brunick, and Cutting’s assertion we need to test a sample of films representing a defined population, and this is precisely what I have done. The following paper demonstrates that the argument that the lognormal distribution is not an appropriate parametric model for the shot length distributions of Hollywood films:
We examine the assertion that the two-parameter lognormal distribution is an appropriate parametric model for the shot length distributions of Hollywood films. A review of the claims made in favour of assuming lognormality for shot length distributions finds them to be lacking in methodological detail and statistical rigour. We find there is no supporting evidence to justify the assumption of lognormality in general for shot length distributions. In order to test this assumption we examined a total of 134 Hollywood films from 1935 to 2005, inclusive, to determine goodness-of-fit of a normal distribution to log-transformed shot lengths of these films using four separate measures: the ratio of the geometric mean to the median; the ratio of the shape factor σ to the estimator σ*=√(2*ln(m/M); the Shapiro-Francia test; and the Jarque-Bera test. Normal probability plots were also used for visual inspection of the data. The results show that, while a small number of films are well modelled by a lognormal distribution, this is not the case for the overwhelming majority of films tested (125 out of 134). Therefore, we conclude there is no justification for claiming the lognormal distribution is an adequate parametric model of shot length data for Hollywood films, and recommend the use of robust statistics that do not require underlying parametric models for the analysis of film style.
Placing this paper alongside my earlier posts testing the lognormality of shot length distributions for Hollywood films prior to 1935 (see here), we can now finally conclude there is no evidence to justify for assuming this model for Hollywood films in general.
Altman DG 2002 Poor-quality medical research: what can journals do?, Journal of the American Medical Association 287 (21): 2765-2767.
De Long J, Brunick KL, and Cutting JE 2012 Film through the human visual system: finding patterns and limits, in JC Kaufman and DK Simonton (eds.) The Social Science of Cinema. New York: Oxford University Press: in press.
Gal I 2002 Adults’ statistical literacy: meanings, components, responsibilities, International Statistical Review 70 (1): 1-51.
Wallman KK 1993 Enhancing statistical literacy: enriching our society, Journal of the American Statistical Association 88 (421): 1-8.
Posted on February 2, 2012, in Cinemetrics, Film Analysis, Film Studies, Film Style, Hollywood, Statistics and tagged Cinemetrics, Film Analysis, Film Studies, Film Style, Hollywood, Statistics. Bookmark the permalink. Leave a comment.