Statistical literacy in film studies II

Previously I have argued that statistical literacy is relevant to film studies because much research on the cinema presents quantitative information in numerical, graphical, and tabular forms, and it is therefore necessary to be statistically literate in order to understand research on film industries, film style, film audiences, and film perception (see here).

This week I want to focus on a further reason for developing statistical literacy in film studies I have touched on briefly before (see here): namely, the ability of film scholars to participate in an evidence–based policymaking process is compromised by a lack of statistical literacy.

‘Evidence-based policymaking’ has become one of the key phrases of the past 15 years, and refers to ‘a policy process that helps planners make better-informed decisions by putting the best available evidence at the centre of the policy process’ (Segone & Pron 2008). Statistics have been described statistics as the ‘eyes’ of policymakers (AbouZahr, Ajei, & Kanchanachitra 2007), while Scott (2005: 40) writes that ‘good policy requires good statistics at different stages of the policymaking process, and that investment in better statistics can generate higher social returns.’ Most people involved in a decision-making process will be using data collected, analysed, and interpreted not by themselves but by professional statisticians, sociologists, market researchers, economists, and so on. It is important to recognise that while we need to be able to understand the information presented to us as part of the making of policy we do not necessarily need to be involved in the research process itself. You can criticise research even if you are not a researcher, and you can criticise statistics in research even if you are not a statistician. It is necessary, therefore, to bear in mind the difference between ‘statistical competence’ and ‘statistical literacy’ I noted in my earlier post.

A distinction can be made between people who are users of statistics and those who are provider of statistics. Whilst it may be unrealistic for professional decision-makers and practitioners to be competent doers of statistics, it is both reasonable and necessary for such people to be able to understand and use statistics in their professional practice. Integrating statistics into practice is a central feature of professions. An increasingly necessary skill for professional policy-makers and practitioners is to know about the different kinds of statistics which are available; how to gain access to them; and, how to critically appraise them. Without such knowledge and understanding it is difficult to see how a strong demand for statistics can be established and, hence, how to enhance its practical application (Segone & Pron 2008).

Participating in a policy making process therefore requires – as a minimum – the ability evaluate research and to understand quantitative information presented in a variety of forms.  The Australian Bureau of Statistics put this very clearly:

The availability of statistical information does not automatically lead to good decision-making. In order to use statistics to make well-informed decisions, it is necessary to be equipped with the skills and knowledge to be able to access, understand, analyse and communicate statistical information. These skills provide the basis for understanding the complex social, economic and environmental dimensions of an issue and transforming data into usable information and evidence based policy decisions.

If you do not understand the information provided to you, the methodologies used, and the pitfalls of both how can you make a sensible decision about which policies have been effective in the past and how can you decide which will provide the best policy for success in the future? Or, as Florence Nightingale wrote, ‘Of what use are statistics if we do not know what to make of them?’

These issues are directly relevant to film studies and its relation to policymaking for film and film education in the UK. The DCMS policy review published in 2012 recognised ‘the need for a strong evidence base for film policy’ and recommended the establishment of a ‘research and knowledge function’ for the BFI in order to

a) collaborate with industry and stakeholders to generate robust information and data on which to base policy interventions, b) assist in the design of BFI policy and funding interventions from the outset to produce learning that can inform future policy, c) actively disseminate results and learning from funding interventions, and d) over time build and maintain a valuable and accessible knowledge base for the benefit of the public, the BFI, Government, industry, academia and all other stakeholders in film.

Evidence-based policymaking has clearly arrived at the BFI, and statistics will inevitably be a part of this process. The BFI’s research outputs already have a substantial statistical component. Obviously, the statistical yearbook is the standout case here, but the Opening Our Eyes report (see here) and the recent policy review both use information presented in numerical, tabular, and graphical forms. These are intended to be used as part of the evidence base for subsequent policy making regarding film education and training (as articulated in the New Horizons document, see here), film distribution, and film production.

Other agencies also produce data-heavy reports. For example, Skillset notes that ‘research provides the evidence, authority and justification for all we do’ and includes large amounts of statistical information in its surveys. There is also much research available from the EU that is loaded with statistics. To these we can add trade publications (Screen International, Variety, etc) and academic research on the cultural economics of film (such as those papers collected together for last week’s post here). Again, this is information that is supposed to provide a basis for decision-making about UK film policy, and all of it containing quantitative information to be used as the desired evidence-base.

The ability to participate in debates is predicated on an assumption that those involved in this process are sufficiently statistically literate to be able to work with the available data and analyses thereof. However, statistical literacy is not a part of the film studies curriculum in the UK at any level. Consequently, film scholars who do not possess the required level of statistical literacy will not be able to fully engage with any evidence-based policy process. Furthermore, film studies courses are not producing graduates with the required skills to participate in debates on film policy in the UK and so this situation will not change. This cuts both ways:

• If you’re not statistically literate, how are you going to know which questions to ask of the information presented to you?
• If you’re not statistically literate, how are you going to communicate your ideas to those with ultimate responsibility for decision-making?

Since the BFI was re-constituted following the abolition the UK Film Council, film studies has to work harder to make its voice heard in the same quarters as industry bodies that have much more experience of lobbying government agencies and are much more effective at it. There is a risk that film studies will be overlooked: for example, in New Horizons ‘education’ tends to be equated with ‘training’ and academic film studies is largely absent, while the panel for the DCMS policy review did not include a single academic working on film in any field let alone film studies. Without taking statistical literacy seriously film studies will find it more difficult to make its voice heard, and risks being reduced to a passive observer of the policymaking process unable to engage in key aspects of the debate because of a lack of relevant skills in understanding the complex and varied dimensions of an issue.

The other side of this coin is that if the BFI is going to produce numerous reports containing large amounts of quantitative information and expects (deep breath) ‘stakeholders’ to participate in an evidence-based policymaking process then it needs to ensure those involved are sufficiently literate to work with statistics. Are film producers statistically literate? Is the Minister for Culture, Communications, and Cultural Industries statistically literate? Is Amanda Nevill statistically literate? The BFI has to take a lead in promoting statistical literacy in order to render consultation processes meaningful, and other film and education bodies have to follow.

The alternative is to have an evidence-based policymaking process in which no-one is able to communicate, understand, and/or challenge the evidence effectively.

References

AbouZahr C, Adjei S, and Kanchanachitra C 2007 From data to policy: good practices and cautionary tales, The Lancet 369 (9566): 1039-1046.

Scott C 2005 Measuring up to the measurement problem: the role of statistics in evidence-based policymaking, in New Challenges for the CBMS: Seeking Opportunities for a More Responsive Role. Proceedings of the 2005 CBMS Network Meeting, Colombo, Sri Lanka, 13-17 June 2005: 35-93.

Segone M and Pron N 2008 The role of statistics in evidence-based policymaking, UNECE Work Session on Statistical Dissemination and Communication, Geneva, 13-15 May 2008.

Death of Screen Yorkshire

On Tuesday 18 January 2011, Screen Yorkshire, the regional screen agency for the Yorkshire and Humber region, announced that it was entering into ‘a consultation process with a number of staff regarding the future of their posts as part of an ongoing review of the future of the organisation.’ The actual announcement can be read here.

Screen Yorkshire’s announcement followed from the previous night’s (17 January 2011) BBC Look North (Yorkshire) which led with the story that the agency had run out of money and was making staff redundant following the government’s decision to abolish the UK Film Council, the regional screen agencies, and the regional development agencies. A contract with Yorkshire Forward worth £10.2 million to promote the screen industries will end in March 2011. Since Screen Yorkshire is a regional screen agency jointly funded by the UK Film Council and Yorkshire Forward, the regional development agency, this was inevitable. The government has proposed that as part of its restructuring of the industry that three major production hubs should be established in the UK as part of ‘Creative England,’ and the one for the North (covering the North East, the North West, and Yorkshire and Humber) is to be located at Manchester.

BBC Look North (Yorkshire) announces that Screen Yorkshire has run out of funding (17/01/11)

According to reports, the ‘restructuring’ will see up to 15 of the agency’s 19 members of staff made redundant.

Screen Yorkshire’s situation is not unique:

• Last December, North West Vision+Media announced that 25 posts were under review out of a total staffing of 35, and that it’s funding beyond March 2011 is also unclear. However, this is less of a problem due to the government’s decision to locate a Creative England hub in Manchester. Staff at Vision+Media are already working four-day weeks following 20% pay cuts in November 2010.
• Screen East went into liquidation last year with debts of £4 million, and this is a region that is close to London and includes Leavesden Studios (where the Star Wars prequels were shot).

Screen Yorkshire has announced that it will continue to deliver its existing contracts over the next year, but it seems likely that it will not be able to function properly as a regional screen agency from April 1 until the creation of the ‘Creative England’ hub for the North. The due diligence process for replacing the UK Film Council with the ‘new BFI’ has not been completed, and no plans for ‘Creative England’ have been released by the Department of Culture, Media, and Sport. Although one round of consultations has apparently been completed on the ‘new BFI’ and Creative England, the next round has not yet begun, while the reality of the recession is overtaking the regional screen agencies. This is not an orderly transition from one policy regime to another, and the fact that we are seeing regional screen agencies running out of funding before the final decisions about the future of film institutions have been taken only reinforces the image of film policy at the DCMS under Jeremy Hunt and Ed Vaizey as incoherent, poorly planned, and incompetently executed.

The future of bodies such as Screen Yorkshire may be in providing regional support outside of the major CE hubs, but this is a return to the days of the old screen commissions of the 1990s when bodies such as the Yorkshire Screen Commission, based in Sheffield, were the only source of contact for producers from both within and without. These commissions had low levels of staffing and were largely cut off from the other bodies responsible for film policy. The introduction of the regional screen agencies was, in part, supposed to remedy precisely this problem. As I have discussed elsewhere, part of the introduction of the regional screen agencies was a process of bureaucratization and professionalization that saw specialized staff hired to fulfil specific roles on a full-time basis in an organized manner. (See Redfern N 2005 Film in the English regions, International Journal of Regional and Local Studies 1 (2): 52-64). Under the government’s announced restructuring of film policy, Yorkshire will have no dedicated film body, with production/funding/distribution funding distributed from Manchester and Film London responsible for promoting locations. If Screen Yorkshire can survive in some form to fulfil this role then that will at least preserve some useful knowledge and key relationships at the regional level. If it cannot, then that bureaucratization and professionalization will be undone.

The impact will not only be felt in the film and television industries, but will also have severe consequences for video games developers in the region. Game Republic is a part of Screen Yorkshire, and the loss of funding to the regional screen agency will lead to the closure of the network unless funding can be secured from private sources.

This story has been followed up by FT.com (here), The Guardian (here and here), The Telegraph and Argus (here), Yorkshire Evening Post (here), The New Statesman (here), The Stage (here), The Drum (here), and Develop (here).

UK film production and the world

In this post I use data from the UK Film Council Research and Statistics Unit to look at the types of films produced in the UK from 2003 to 2009, and the connections between UK film production and other parts of the world. Data was collected for a total of 996 films. Documentaries were excluded from the sample. Note also that this data only includes films with a budget of £500,000 or greater, and so only provides a partial picture of UK film production.

The UK Film Council groups films into five categories: co-productions (COP), incoming co-productions (ICP), domestic features (DOM), inward productions (INW), and films that come to the UK to source visual effects (VFX). Here I only use the first four categories because data for VFX has only been collected since 2007; and I combine COP and ICP into a single category, as the frequency of the latter is low for 2003 to 2006 and zero for 2007 to 2009. The percentage of each category of film is presented in Figure 1, which also includes the data table.

Figure 1 Film production in the UK by type of film, 2003 to 2009

From Figure 1 we can see that there are two clear trends. First, the percentage of co-productions has fallen by two-thirds from 60% to 20%. Second, the percentage of domestic productions has increased by a factor of three, from approximately 20% to 60%. This change occurs in two steps, with an initial step occuring at 2005, with the final change in 2007. This coincides with the initial proposal of the cultural test for British films in 2005 and its final implementation in 2007, and cannot therefore be attributed to the global financial downturn that began in 2007/2008. The percentage of inward features appears to be immune from this change, with the notable exception of 2005 which shows a small increase.

From looking at the actual counts in Table 1 It is clear from this data that there has been an increase in domestic productions with the introduction of the cultural test, but that this has not replaced the number of films lost from the decline of co-productions, and that overall the number of films produced in the UK has decreased. This data does not of course tell us anything about the level of production spending in the UK, and this is obviously a crucial factor in considering the health of the film industry in the UK.

Table 1 Frequency of British films produced by category, 2003 to 2009

Nonetheless, this data should give cause for concern because it indicates that film production in the UK has become increasingly one-dimensional. A healthy film industry is one that can absorb shocks to the system as patterns of production in the global film industry change – but if film production becomes too concentrated into a single class of films this increases the vulnerability of the industry to a crisis. We might say that the cultural test has been successful in stimulating the British film industry, but that this has come at the expense of the film industry in the UK. (Here I make a distinction between all film production activity that takes place in the UK – the UK film industry – and that part of this production activity that is defined as culturally British – in other words, the ‘British national cinema’). A sudden drop in funding for culturally British films would plunge the UK film industry into a crisis of production, which would not be able to make up the short fall from productions originating in other parts of the world. The tax incentives available for film production in the UK are therefore of great importance, and without them we would likely return to the low numbers of production last seen in the 1980s.

It is perhaps instructive to think of the cultural industries in ecological terms: they are a system in which the companies are subject to forces of competition, predation, and extinction, and a change in one part of the system can have very significant consequences throughout the system as a whole. Biodiversity is one of the measures of the health of an ecological system, and economic diversity should also be thought of as a measure of the health of the cultural industries. The above data suggests that the economic diversity of the UK film industry has declined over the past seven years, as film production has become over-dependent on a single class of films. Disrupting the delicate equilibrium of this system could have significant consequences.

• If a government were to announce it was disbanding the government body responsible for developing and implementing film policy and for distributing funding for film production in the UK without announcing what would take its place, thereby reducing the ability of producers to attract funding because of the uncertainty such a decision would introduce, could also have a negative impact on the level of film production.
• A determination to reduce immigration from non-EU countries, for example, would not only reduce the number of scientists coming to the UK but could also have a negative impact on the film industry as filmmakers from outside the UK decide to make their films in more welcoming countries.

The first of these has already happened, and threatens to disrupt levels of investment in film production in the UK. The second will be introduced next year, and may further reduce the diversity of film production in the UK. Either one of these could create problems, but both together indicate a lack of foresight on the part of the coalition government.

Part of the drop in co-productions has been attributed to the fact that some films that would previously have been classed as co-productions are now able to qualify as ‘British’ under the terms of the cultural test, and thereby enjoy the full benefit of being a ‘qualifying British film.’ This argument has been put forward by John Graydon, who was involved in structuring the UK’s film tax credit system, and who notes that this represents the success of incentivising film production in the UK (Mansfield 2009). While this may certainly be a fair assessment of the status of some films, it cannot account for the full-scale of the decline in co-productions and does not explain why there has been an overall decline in the total number of productions for each year.

We can also evaluate the diversity of the UK film industry by looking at how the UK is connected to the rest of the world. In Table 2, we have the number of films that have a connection to one or more other countries that were produced in the UK, and we see the same pattern of decline noted above. This table includes films from all the UK Film Council categories.

Table 2 Films produced int he UK connected to at least one other country, 2003 to 2009

These 642 films account for a total of 870 connections to 49 different countries. The number of connections exceeds the number of films because a film may have connections to more than one country. (No film has more than five non-UK co-production partners listed, and most films are productions that involve a UK producer and a producer from one other country). A year by year breakdown by country is presented in Table 3 (NB: this table is quite large).

Table 3 Number of connections to co-producing countries for films produced in the UK, 2003 to 2009

The USA is consistently the most important source of connections to the UK with France a distant second, but where the US has remained at the top of the pile the frequency of connections to France has fallen sharply. Interestingly, India is listed as the third largest source of connections beating Germany into fourth place. ike France, Germany has gone from being a major production to partner to an occasional source of connections after 2004. This is also true of Canada, which has no connections listed for 2009 at all, as well as Spain, Italy, Ireland, Luxembourg, and Denmark. The category ‘Europe’ is not defined by the UK Film Council.

If we look at the diversity of countries connected to the UK in this way and the number of connections, we find that European countries are the most numerous at 33, accounting for a total of 528 connections (I include Turkey as a European country here). North america accounts for only 3 countries but the USA accounts for 173 connections and Canada 60 so clearly. Next comes ASia, which accounts for only four countries but a total of 75 connections. It should be noted, however, that almost all of these are accounted for by India. Notable absentees from the list of co-production partners in Table 3 are China, Japan, and South Korea. Africa is represented by four countries totalling 15 connections, of which South Africa account for two-thirds. Oceania is represented by Australia and New Zealand, providing a total of 15 connections; while the only South American countries included are Argentina and Brazil, accounting for only 3 films. Figures 2 and 3 make this information somewhat easier to appreciate.

Figure 2 Countries with connections to UK film production by region, 2003 to 2009

Figure 3 Number of connections to countries sorted by region, 2003 to 2009

Often when we talk about globalisation in the film industry we imply something that happens all over the world, but this is clearly not the case for connections between the UK film industry and elsewhere. The places to which film production in the UK is connected can be sorted into three major groups: first, there are the countries geographically closest to the UK – i.e. Europe; second, there are the countries that are historically closest to the UK – i.e. former colonies such as India, Canada, Australia, and South Africa; and, third, there is the United States, which is the dominant global power in the film industry. We can therefore say that some of the ways in which the UK film industry is globalised are through proximity, legacy, and domination by a superior market. When we turn our attention to countries that are far from the UK, that do not have a close historical/cultural relationship, and which are not major world cinema powers – in other words South America and the Far East – we find there are very few connections, if any. In this respect the UK is not that different from Poland, Malaysia, Chile, or Morocco that I have looked at elsewhere on this blog (see here and here).

Given that it is countries such as Brazil, Russia, India, and China that are tipped to become major global economies in the 21st century, it is imperative that the diversity of feature film production in the UK can be expanded to include links to these countries. India aside, one of the greatest challenges facing policy makers in the UK is how to go about establishing connections to these distant places in the absence of a strong historical relationship. It is difficult to see how this will be achieved whilst restricting immigration from non-EU countries.

The proposed reduction in immigration will hit Indian and American filmmakers hardest, and yet beyond Europe these are the only significant sources of inward investment and co-production partnerships for the UK film industry. Without them, the UK will become increasingly more dependent upon the EU. And yet, as I made clear above, the introduction of the cultural test for British films has dramatically reduced the number of co-productions between UK producers and their European counterparts. A further loss of diversity will only increase the vulnerability of the UK film industry to crises it has been historically ill-prepared to deal with.

References

Mansfield M 2009 A Report on the British Film Industry for Shadow DCMS. This report can be accessed here.

Some notes on Cinemetrics III

This post was originally intended to be part of a thread on the discussion board of the Cinemetrics website, but for some reason it did not upload properly. This post presents my piece in full, but readers should refer to the original thread to get the preceding parts of the discussion.

The difference between the median/mean ratio for two film or two groups of films (e.g. silent films and sound films) can be explained by the presence of outliers in the data and the influence they have on the mean shot length. This can be demonstrated by looking at the shot length distributions for the two versions of Blackmail, The Lights of New York, and Scarlett Empress. (The data for these films can be found in the Cinemetrics database).

Imagine you have two data sets that are identical except for a single value. For example,

A:  1, 2, 3, 4, 5, 6, 7, 8, 9, 10

B:  1, 2, 3, 4, 5, 6, 7, 8, 9, 20

For data set A, the median is 5.5, the mean is 5.5, and so the median/mean ratio is 5.5/5.5 = 1.0. For data set B, the median is 5.5, the mean is 6.5, and the median/mean ratio is 5.5/6.5 = 0.85. The changes in the mean and the median/mean ratio are due to the influence of a single outlying data point, and do not reflect the fact that the two data sets are otherwise identical.

This is precisely what we see when we look at the two versions of Blackmail. In the table below, we have the mean shot length, the median shot length, and the ratio of the median to the mean.

	Blackmail (silent)	Blackmail (sound)
Median shot length (s)	5.6	5.1
Mean shot length (s)	8.1	10.4
Median/mean	0.69	0.49

Looking at the mean, we might think that the impact of sound technology was to lead to a change in style, with an increase in shot lengths (the difference in the means is 2.3 seconds). However, we know that shot length distributions are positively skewed with outlying data points, and that the mean is, therefore, problematic. The difference in the medians is small (only 0.5 seconds), indicating that no such change occurred. This conclusion is supported by a medians test, which shows no significant difference: p = 0.135. A more complete picture may be obtained by looking at the five number summary for each film.

	Blackmail (silent)	Blackmail (sound)
Minimum shot length (s)	0.1	0.1
Lower quartile (s)	2.9	2.5
Median shot length (s)	5.6	5.1
Upper quartile (s)	10.1	11.5
Maximum shot length (s)	104.3	144.6

Looking at this data, we would conclude that the difference between the styles of these two films occurs above the upper quartile – the difference is in the length of the outlying data points away from the mass of the data. The lower quartiles in the silent and sound versions are similar – each film has approximately the same proportion of shots less than or equal to 2.9s (25% and 29%, respectively). This is also the case for the medians: half of the shots in the silent version are less than or equal to 5.6 seconds, while half the shots in the sound version are less than or equal to 5.1 seconds. The difference between the upper quartiles is greater (1.4 seconds) but is still less than the difference between the means – where 75% of the shots in the sound version are less than or equal to 11.5 seconds, this proportion in the silent version is 79% for the same value. In fact by looking at the empirical cumulative distribution functions for both versions of Blackmail (see Figure 1) it is clear that they have almost identical distributions; and a 2-sample Kolmogorov-Smirnov test shows that there is no statistically significant difference for any shot length in the two versions of this film (D = 0.0666, p = 0.1881). (Note that the distribution functions in the graph below are empirical – i.e. they are the actual probability distributions of the shot length data from the Cinemetrics database, and they are not theoretical distributions).

Figure 1 The empirical cumulative distribution functions for the two versions of Blackmail (1929)

The only explanation for the difference in the means, and for the difference in the median/mean ratios, is the influence of the outliers on the mean. Using the mean – or any statistic based on the mean – will lead to incorrect conclusions. Using the mean, we might conclude that the shot lengths in the two versions of Blackmail show a statistically significant increase with the use of sound technology – but this would be wrong. As in the example data sets above, the difference we find in the median/mean ratios reflects the influence of these outlying data points, and does not accurately reflect the distribution of shots in the two versions in the two versions of Blackmail. When we use measures of dispersion that are robust against outliers we do not see the large difference in the dispersion of shot lengths we would expect with the median/mean ratio. The median absolute deviation for the silent version is 3.3 seconds and for the sound version is 3.1 seconds; while the interquartile ranges are 7.2 and 9.0 seconds, respectively.

Let us now look at the shape factors of the two versions of Blackmail. The standard deviation of the logarithms of the shot lengths for the silent version is 0.91; and the equivalent value for the sound version is 1.05. This suggests that the two distributions have different shapes – the sound version being more widely dispersed than the silent version. However, from the five number summary, the empirical cumulative distribution functions, and the robust measures of dispersion we can see that this is not the case. Therefore, we have two versions of the same film with shot length distributions that show no statistically significant difference, but with a large difference in the median/mean ratio and a corresponding difference in the lognormal shape factors. These differences are due to the influence of outlying data points, and do not accurately reflect the nature of the relationship between these two distributions.

A second example can be used to illustrate what happens when we have two films with the same mean but different median shot lengths. The mean shot length, the median shot length, and the ratio of the median to the mean for Lights of New York and Scarlett Empress are presented in the table below.

	Lights of New York	Scarlett Empress
Median shot length (s)	5.1	6.5
Mean shot length (s)	9.9	9.9
Median/mean	0.52	0.66

Looking at the mean shot lengths, we can see that they are identical and we might conclude that these films are cut equally quickly; but looking at the medians we can immediately see that there is a difference of 1.5 seconds, which alerts us to the possibility that Lights of New York is cut quicker than Scarlett Empress. A medians test tells us that there is, in fact, statistically significant difference in the medians of these two films: p = 0.0007.

Now, the median/mean ratio is a crude measure of the dispersion of skewed data set, and the smaller the value of this ratio the more dispersed the data (i.e. the greater the distance between the median and the mean). For symmetrical distributions the median and the mean are equal and the ratio is one; but as shot length distributions are positively skewed the mean will always be greater than the median and the ratio will always, therefore, be less than one. (The ratio is typically given in economic text books as the mean divided by the median [it is used as a measure of income inequality], but this is just the reciprocal of the median/mean ratio). Clearly, the use of the mean as a measure of central tendency will lead us to an incorrect conclusion about the difference in style of Lights of New York and Scarlett Empress; but does it fare any better as an indicator of which film has shot lengths that are more dispersed?

According to the table above, Lights of New York has a smaller median/mean ratio (0.52) and so we would expect the shot lengths for this film to be more dispersed than those of Scarlett Empress (0.66). The standard deviation for Lights of New York is 14.5 seconds, and for Scarlett Empress it is 9.6 seconds – again indicating that the former is more dispersed than the latter. (The lognormal shape factors for these two films are 0.93 and 0.88, respectively). However, when we look at the median absolute deviation and the interquartile ranges we get a different picture.  For both statistics, it is evident that Scarlett Empress is, in fact, more dispersed.

	Lights of New York	Scarlett Empress
Median absolute deviation (s)	2.6	3.5
Interquartile range (s)	7.2	9.3

This can be easily seen when looking at the box plots of these two films (Figure 2). In the box plots note that the interquartile range (the box) for Lights of New York is narrow than that for Scarlett Empress; and that the distance between the minimum shot length (the end of the error bar to the left) and upper inner fence (the error to the right of the box at Q3+(IQR*1.5)) is less for the former (0.9 – 20.9 seconds) than it is for the latter (0.3 – 26.9 seconds). Anything above the greater of these values (i.e. 20.9s and 26.9s is classed as an outlier, while ‘very extreme’ values are defined as Q3+(IQR*3).

Figure 2 Box plots for shot lengths in Lights of New York (1928) and Scarlett Empress (1934)

It is evident, therefore, that (1) shot lengths in Scarlett Empress are more dispersed than those of Lights of New York; and (2) the reason the mean shot length, the median/mean ratio, and the standard deviation give misleading results is because of the influence of the outlying data points in Lights of New York (which account for only 9.76% of this film’s data).

Is the median/mean ratio still useful for estimating the median for these four films? For the silent version of Blackmail the estimated median based on the shape factor given above is 5.3 seconds; and for the sound version is 5.9 seconds. Thus, the first estimate is good with an error of 0.3 seconds (or 4.7%); while the second estimate is less good and is out by 0.8 seconds (16.1%). For Lights of New York, the estimate of the median is 6.4 seconds – an error of 1.3 seconds or 25.1%! The estimate for Scarlett Empress is much better at 6.7 seconds, and is out by only 0.2 seconds (3.7%). We can see, therefore, that estimating the median as the mean/(exp(0.5*(σ^2))) may produce very good estimates but may also produce very bad ones.

The mean is not a robust statistic, and is vulnerable to two factors: the presence of outliers in a data set and the asymmetry of a data set. Unfortunately, these are precisely the characteristics of the distribution of shot lengths in a motion picture. Any value calculated using the mean (e.g. the standard deviation, the median/mean ratio) will not accurately reflect the style of a film due to the impact of outlying data points on the mean. Use of the mean will, therefore, leads us to make a range of incorrect conclusions.

• In the case of the example of Blackmail, we would have incorrectly concluded that there is a difference between the shot lengths of the two versions of this film, when in fact there is no such difference.
• In the example of Lights of New York and Scarlett Empress, we would have incorrectly concluded that there is not a difference between the shot lengths of these two films, when in fact there is such a difference.
• In the example of Lights of New York and Scarlett Empress, we would have incorrectly concluded that the shot lengths of the former are more widely dispersed than in the latter, when in fact the opposite is true.
• Using the mean may produce wither very good estimates of the median or it may produce very bad estimates of the median. Simply relying on this method to lead us to reliable conclusions will not work: if we used the estimate of the median for the sound version of Blackmail in a study we would be basing our analysis on a fundamental error.

The mean shot length is not a reliable statistic of film style. The median/mean ratio suffers from precisely the same problem that has always existed with the mean. It is just a different way of presenting it.

(I’m currently looking at the impact of sound on Hitchcock’s style and Blackmail in more detail, and I’ll put up a post on this subject at a later date. I began working on this piece a couple of months ago and the data I have been using (and the data referred to above) was submitted in 2006 by Isobel Walker. Charles O’Brien has recently submitted new data for the sound version of Blackmail, but I have not looked at this in detail yet).

Measures of central tendency in Cinemetrics

Barry Salt has made several assertions about the nature of film style and the use of statistical methods in the analysis thereof. Chief among these are:

• That the mean shot length accurately describes the distribution of shot lengths in a motion picture.
• That the lognormal distribution is a ‘ruling distribution’ of film style.
• That a shape factor of 0.9 is characteristic of shot length distributions, or that it lies in the interval 0.7 to 0.9.
• The median shot length can be estimated as either equal to mean * 0.6 or equal to mean / exp(0.5σ²), where σ is the shape factor.

Here I subject each of the above claims to scrutiny by testing them against specific data sets for defined populations, and I examine the methodology proposed by Salt in detail.

The mean shot length as a statistic of film style

Salt (1974) proposed that the mean shot length be used as a statistic of film style for shot length distributions. However, the distribution of shot lengths in a motion picture is typically characterised by two features: (1) it is positively skewed, and (2) there are a number of outlying data points that are far from the mean. (I say typically because there is no reason why the shot lengths of a film could not be distributed normally or negatively skewed, but I have not come across such a film). While the mean is the best measure of location for distributions that are symmetrical or near-symmetrical, it is a poor statistic when the data is asymmetrical (i.e. the data is skewed). The mean is not a robust statistic. When we say that a statistic is ‘robust,’ we mean that it is not influenced by data points that are very different from all the rest (outliers). The mean is very sensitive to such outliers and this can pull the mean away from the centre of the data creating a skewed data set: it has an asymptotic breakdown point of 0.0. The asymptotic breakdown point is a measure of the proportion of the data that can be given arbitrary values before the statistic becomes arbitrarily bad (Geyer 2006). So for the mean, the proportion of outlying data points in a sample that the mean can cope with is zero – just a single outlier can wreck the mean as a measure of central tendency, and in the distribution of shot lengths in a motion picture we can expect to find many outliers.

The asymmetrical nature of shot length data also limits the type of statistical tests that can be employed in statistical analysis [1]. Many tests require the assumption (amongst others) that the data is normally distributed. Tests which require an assumption to be made about he underlying probability distribution of the data are called parametric tests. However, for skewed data sets with a number of outliers, the assumption of normality does not hold, and the results of employing parametric tests will be unreliable. As no data is actually distributed normally, small deviations from the true normal distribution can be tolerated, but when we are dealing with shot length distributions we find that the deviations from normality are very large.

These problems may be overcome by using the median shot length as a statistic of film style. the median locates the middle of a data set by dividing the data in two, so that half the data is equal to or less than the median and half is equal to or greater than the median. The median locates the centre of any data set irrespective of shape, and is a more robust statistic than the mean: it has an asymptotic break down point of 0.5.

As we will be unable to use parametric statistical tests to analyse the data, we can turn to nonparametric tests which do not require the same assumptions to be met. Such tests, for example, make no assumption about the distribution of the data, and are often referred to as distribution-free tests. Thus, the two-sample independent t-test requires that the data in both samples be normally distributed; this is not the case for its nonparametric equivalent, the Mann Whitney U-test. This does not mean, however, that nonparametric statistics are assumption-free tests, and it is still necessary to make sure that the assumptions of each test can be met (e.g. that data is independently and identically distributed). As they require fewer assumptions about the nature of the data nonparametric tests are less powerful than their parametric equivalents, but when the requirements for the parametric tests cannot be met they provide a much better alternative than no analysis at all.

The mean shot length should never have been suggested as a statistic of film style. The median is a far superior statistic given the skewed nature of shot length distributions, and analysing film style using this measure of central tendency will provide results that are far more reliable than if we used the mean.

Do motion pictures have lognormal shot length distributions?

An alternative to using the median shot length is to apply a transformation to data in order to remove the skew from the data. Such transformations include raising the value of a data point to a power or taking the reciprocal of a data point, and there are many others. A common transformation employed is to take the logarithm of a data point, which may produce a lognormal distribution. A random variable (X) is said to be lognormally distributed if its logarithm (log [X]) is normally distributed. If the shot length data for a film is lognormally distributed then we have an advantage over the median shot length, as knowing the underlying probability distribution will allow us to use the more powerful parametric statistical tests. However, just as the parametric tests are unreliable when the assumption of normality is not, the same is true if the assumption of lognormality is not met.

The lognormal distribution and the mean shot length

Salt proposes that (1) the mean shot length is the appropriate statistic of film style and that (2) the shot lengths of a motion picture are distributed lognormally (2006: 389-396). However, these are conflicting claims. If the mean shot length was a reliable statistic of film style then we would not need the lognormal distribution: the reason we apply a logarithmic transformation to the data is because the mean does not provide a robust measure of the central tendency of the data in its original scale. Applying a logarithmic transformation to data allows us to recover the symmetry of the data, which we would not need to do if it were already symmetrical. To be specific, it is the arithmetic mean shot length that is incompatible with the lognormal distribution. The arithmetic mean is what we normally refer to when we say ‘mean:’ it is the sum of the data points divided by the number of data points in the sample. The arithmetic mean is NOT the measure of central tendency of the lognormal distribution. If we wish to locate the centre of the lognormal distribution then we use the geometric mean.

Logarithms are useful because they make complicated procedures like multiplication into simple operations like addition. For example, instead of multiplying two numbers together we can simply add their logarithms and by transforming the result back into the original scale we have our answer:

a * b = c ,

is the same as

log (a) + log (b) = log (c).

It does not matter which logarithm you use so long as you are consistent and use the correct method to back-transform the result. The two main logarithmic transformations are the common logarithm (log₁₀ [X]), which uses base 10; and the natural logarithm (ln [X]), which uses base e.

Now, if we transform the length of each shot in a film (X) into its logarithm (log [X]), we can calculate its average in the usual way – that is, we add up all the logarithms and divide by the number of shots. Transforming the average of the logarithms back into the original scale will give us the geometric mean. In the original scale, this is the equivalent of multiplying all the shot lengths together and then taking the nth root, where n is the number of shots in the film. As we are using multiplication instead of addition, the geometric mean of lognormally distributed data is clearly going to be very different from the arithmetic mean. In fact, the geometric mean of a data set will always be less than the arithmetic mean (unless all the data points are equal).

In Table 1, the shot length data (X) for A Busy Day (Charles Chaplin, 1914) is presented along with the natural logarithm of each shot (ln [X]). (This film has a shot length distribution that is lognormally distributed – Shapiro-Wilk: w = 0.9783, p = 0.6556). The film is 340.4 seconds long and includes 38 shots (once title and dialogue cards have been removed). The arithmetic mean shot length of A Busy Day is 340.4 divided by 38, and equals 9.0 seconds. The sum of the logarithms is 65.4763, and dividing this figure by 38 gives 1.7231. Transforming this figure back into the original scale gives us a geometric mean of 5.6 seconds.

TABLE 1 Shot length data in the original scale (X) and its natural logarithm ln (X) for A Busy Day (1914)

One of the reasons people get confused with the lognormal distribution is because, unlike the normal distribution, its expected value (E[X]) and its measure of central tendency are not the same. (For the normal distribution, the expected value and the measure of central tendency are both the arithmetic mean). The expected value of a Lognormal distribution is equal to the exponentiate of the geometric mean plus half the variance: E[X] = exp(μ+0.5σ²). For A Busy Day, the mean of the logarithms (μ – the geometric mean) is 1.7231 and the variance (σ²) is 1.0259. If we add the geometric mean to half the variance (1.7231 + 0.5129) we get 2.2360. Transforming this value back to the original scale (the exponentiate) we get 9.4 seconds, which is approximately equal to the arithmetic mean shot length, and we know this does not locate the centre of the shot length distribution for this film. For a Lognormal data set, the geometric mean will be approximately equal to the median. We know that the median will locate the centre of any data set, and for A Busy Day the median shot length is 4.9 seconds – much closer to the geometric mean than the arithmetic mean. (Finding the median of the logarithms (1.5813) and then converting back to the original scale is equal to the median shot length. Using the median of the logarithms gives a poorer estimate of the expected value (8.1 seconds) than the geometric mean). For a discussion see Olsson (2005).

The shot length data for A Busy Day is presented above, so you can do this for yourself. NB: I calculated the variance for this film using Microsoft Excel’s function for finding the population variance (=VARP(array), where the denominator is n), while using the sample variance function (=VAR(array), where the denominator is n-1) will give E[X] = 9.5 seconds [2]. It should also be noted that, while Excel 2007 has a function for calculating the geometric mean (=GEOMEAN(array)), this will only work for up to 255 data points (and for earlier versions of Excel considerably less). Transforming each shot length to its logarithm and then finding the average will work for any sample size in Excel.

It is clear that the arithmetic mean shot length should never have been proposed as a statistic of film style. If, as Salt claims, shot lengths are lognormally distributed resulting from the multiplication of independent factors, then it is necessary to use a multiplicative central tendency and this is the geometric mean shot length. It is necessary to choose between two claims: are we going to claim that the arithmetic mean shot length is the best statistic of film style or are we going to claim the shot lengths are lognormally distributed? Since we already know that the arithmetic mean is not a robust statistic for skewed data sets and we have already resorted to using the logarithms of the data, it would seem obvious to jettison the arithmetic mean and to use the geometric mean as our statistic – assuming, of course, that the shot lengths of a motion picture are lognormally distributed.

This would, of course, mean that every time the arithmetic mean shot length has been quoted as a statistic of film style, it is simply wrong.

Is the assumption of lognormality justified for the distribution of shot lengths in a motion picture?

In Salt (2006: 389-396), the ‘generality of the Lognormal distribution for shot lengths in movies’ is asserted but not demonstrated. Examples of some films that are claimed to have Lognormal shot length distributions are featured alongside some films for which this claim is not made, but the extent to which these claims can be generalised is in unclear. Salt admits that the sample in this study is not representative due to the presence of a number of films with very large mean shot lengths [3], and so on what is the claim that a lognormal distribution can be usefully used to model shot length distributions based? We do not know from which population the sample is drawn or what the sample size is. A further problem is that it is not clear what Salt defines as a film in which the shot lengths are lognormally distributed. The coefficient of determination is presented as a measure of goodness-of-fit, but there is no decision rule stated as to what value of R² can be considered ‘good.’ We do not actually know from this if the lognormal distribution is reliable enough to use in the analysis of film style, because we do not know how common it is for films to have lognormal distributions. Despite these problems, Salt has since made a much stronger claim that the lognormal distribution is a ‘ruling distribution’ of film style [4]. This claim assumes that at least a majority of films will have shot lengths that are lognormally distributed, although this has not yet been demonstrated.

Other probability distributions have been used in modelling shot lengths. Fujita (1989), for example, surveyed 32 educational television programmes and found that an Exponential distribution provided a good fit for the shot lengths in 30 cases. The Weibull, Gamma, and Poisson distributions (amongst others) have all been proposed as the best model for the shot lengths of motion pictures (Cotsaces et al. 2009, Taskiran and Delp 2002, Truong and Venkatesh 2005, Vasconcelos and Lippman 2000). Indeed, Salt (1974) used the Poisson distribution to model shot lengths, and also found films in which this hypothesised distribution did not hold.

It is a simple matter to estimate the proportion of films with a Lognormal distribution, and this is experiment is conducted below.

Sample

The samples used are the fifty films that I analysed earlier in my study of the impact of sound technology on the median shot lengths in Hollywood cinema. These films are divided into two samples: silent films produced between 1920 and 1928 inclusive (n = 20); and sound films produced from 1929 to 1931 inclusive (n = 30). The descriptive statistics for each film can be found by referring to my earlier paper.

Method

The shot lengths of each motion picture in the samples are transformed into their natural logarithms. The lognormality of the data is then tested using the probability plot correlation coefficient (PPCC) employing a Blom plotting position, with a significance level of 0.05 (Looney and Gulledge 1985, see my earlier post on how to do this). Where the PPCC for a film was just under its critical value, the result is checked using a Shapiro-Wilk test (α = 0.05).

The proportion of films with lognormally distributed shot lengths is then calculated, along with an approximate 95% confidence interval using the adjusted Wald method (Agresti and Coull 1998). This will be our estimate of the proportion of films that have lognormal distributions for the populations from which the samples are drawn.

Calculations were performed using Graphpad online calculators and PAST 1.89 (2009). The critical values for the PPCC can be accessed at the NIST website.

Results

The results of the PPCC test for the silent films are presented in Table 2, and for the sound films in Table 3. Only one film needed to be checked using the Shapiro-Wilk test: Behind the Make-up is not lognormally distributed (w = 0.9873, p = 0.0184).

TABLE 2 Sample size and PPCC (α = 0.05) for silent films produced in Hollywood, 1920 to 1928 inclusive

TABLE 3 Sample size and PPCC (α = 0.05) for sound films produced in Hollywood, 1929 to 1931 inclusive

Of the twenty silent films, six have Lognormal shot length distributions, and the proportion of silent films produced in Hollywood from 1920 to 1928 inclusive with a Lognormal distribution is estimated to be 0.30 (0.14, 0.52).

Of the thirty sound films, thirteen have Lognormal shot length distributions, and the proportion of sound films produced in Hollywood from 1929 to 1931 inclusive with a Lognormal distribution is estimated to be 0.43 (0.27, 0.61).

While some films do have shot lengths that are lognormally distributed, Salt’s statement that the lognormal distribution is a ‘ruling distribution’ of film style cannot be justified. In fact, in neither sample is there a majority of films with a lognormal distribution. If an analysis of film style is conducted using the assumption of then it is likely that the results will be unreliable.

The geometric mean is a superior measure of central tendency for skewed data sets with lognormal distributions. However, as no evidence has been presented that would justify the assumption that shot lengths are lognormally distributed the use of the geometric mean is questionable. Again, the median shot length is available as an alternative that can be used reliably as it locates the centre of a distribution as the middle ranked value in a data set, and does not rely on an underlying probability distribution.

Do shot length distributions have a characteristic shape factor?

Each theoretical distribution is described by a set of parameters. The Lognormal distribution is described by the parameters and the shape factor, σ. Salt has claimed that the characteristic shape factor for the Lognormal shot length distributions of a motion picture is ~0.9 [5]. The relevance of this claim is lessened by the fact that there is no evidence to justify the claim that shot lengths are lognormally distributed. This claim is different to the one made in Salt (2006: 393), where it was asserted that the shape factor will lie in the interval 0.7 to 0.9.

Again, it is a simple matter to test both these claims.

Hypotheses

The first research question we are addressing here is ‘the lognormal shape factor of a shot length distribution is 0.9.’ The statistical hypothesis is:

• H₀: the shape factor (σ) = 0.9

The second hypothesis we will address is the claim that ‘the lognormal shape factor of a shot length distribution will lie in the interval 0.7 to 0.9.’

Sample

The two samples of Hollywood films used above are employed in this test.

Method

The shape factor for each film is determined by maximum likelihood estimation (MLE) for the lognormal distribution. The mean value of σ for each data set is then calculated, and compared to the hypothesised value of 0.9 using a one sample t-test. A p-value of less than 0.05 is considered significant. The proportion of films with σ in the range 0.7 to 0.9 is then calculated, along with an approximate 95% confidence interval using the adjusted Wald method

MLE is performed using online calculators (Wessa 2008), and the t-test is performed using Microsoft Excel 2007. Graphpad online calculators were used to produce the confidence intervals for the proportions.

Results

The Lognormal shape factor for each film is presented in Table 4 for the silent films and Table 5 for the sound films.

TABLE 4 Lognormal shape factors for silent films produced in Hollywood, 1920 to 1928 inclusive

TABLE 5 Lognormal shape factors for sound films produced in Hollywood, 1929 to 1931 inclusive

The mean shape factor of the silent films is 0.7437 (SD = 0.0617), and is significantly lower than the hypothesised value of 0.9, t (19) = 11.3303, p = <0.0001.

The mean shape factor of the sound films is 0.9411 (SD = 0.1066), and is significantly greater than the hypothesised value of 0.9, t (29) = 2.1134, p = 0.0433.

If we take Salt’s alternative claim that σ will lie in the interval 0.7 to 0.9, then we can say that for the silent films this is a much more useful estimate, with a proportion of 0.70 (0.48, 0.86) in the specified interval. For the sound films, however, it is less good with a proportion of 0.47 (0.30, 0.64).

The hypothesised shape factor of 0.9 is not a good estimate for either sample, while the specified range of 0.7 to 0.9 is only a reasonable estimate for the silent films and even then we can expect over one-quarter of the films to lie outside this interval.

The claim that there is a characteristic shape factor for the distribution of shot lengths in a motion picture is not supported by the evidence.

When is a mean shot length not a statistic?

There are clearly serious problems in using the arithmetic mean shot length as a statistic of film style, and  Salt has tried to shift the justification for keeping the mean shot length to the argument that it can be used to estimate the median shot length [6]. To further add to the confusion of using the arithmetic mean with the lognormal distribution, we now have the claim that the mean shot is both the desired statistic of film style and is desirable as a means of estimating the median. Why, if the mean shot length is the statistic we desire, do we need these methods of estimating the median? Why, if the median can be estimated from the mean, has no one ever used this estimated median to describe changes in film style? As before, it is a question of competing claims: it is either the mean or the median, as they are different for skewed data sets, and not both. As it is a simple matter to demonstrate that the mean shot length is not a robust statistic, then it should be disposed of. Again, if the mean shot length is not the desired statistic of film style, then it would be necessary to admit that every time the mean shot length has been quoted in books and journal articles, this was wrong.

This is all very well, but it begs a fundamental question: is the estimated median any good?

Can the median shot length be reliably estimated from the mean shot length?

Salt proposes two methods for estimating the median shot length from the arithmetic mean shot length, which, for the sake of simplicity, I shall refer to as Method A and Method B:

• Method A: median = mean * 0.6
• Method B: median = mean / exp(0.5σ²), where σ is the shape factor.

This two methods should produce approximately the same results when σ = 0.9.

Again this is simply an assertion and Salt provides no data or results to back up this claim.

Sample

The two samples of Hollywood films used above are employed in this test.

Method

For clarity, the following symbols are used:

• Med is the true value of the median shot length.
• Med_A is the estimate of the true value of the median using Method A.
• Med_B is the estimate of the true value of the median using Method B.

As Salt claims that σ = 0.9, using Method B is immediately problematic as I have already demonstrated that this is not a good estimate of the shape factor of the films in the two samples. In order to allow for this Method B is used twice – once where σ = 0.9, and once where σ is the MLE-derived value in Tables 4 and 5.

The value of Med_A or Med_B is considered a good estimate for Med if it is included in the 95% confidence interval of Med. Note that this is not the same as saying that Med_A or Med_B will be equal to Med – only that they will estimate Med if they lie within an interval with a specified confidence level. These methods will therefore introduce some error into any analysis even when they are good estimates, but this error will be known.

The confidence intervals for the median were calculated using the binomial method. It is important to remember that while the shot length data itself is not binomially distributed, the median shot length is determined by its rank in the ordered sample. Therefore, when we calculate the confidence interval for a median were apply the binomial method to the ranks of the ordered data and then transpose this on to the ordered data – i.e. calculate the rank of the lower (j) and upper (k) limits of the interval for the proportion 0.5 and then the shot lengths that are ranked jth and kth in the ordered data. The binomial method is NOT applied to the shot lengths themselves. Using the binomial method tends to produce a conservative interval, but all the intervals are at least 95% and no film has a confidence interval greater than 96.41% [7]. See Curwin and Slater (2008: 296) for a simple introduction on how to do this and the large sample approximation.

The proportion of good estimates is calculated, along with an approximate 95% confidence interval using the adjusted Wald method.

Results

The results for the sample of silent films are presented in Table 6, and for the sound films in Table 7.

TABLE 6 Median estimation for silent films produced in Hollywood, 1920 to 1928 inclusive

TABLE 7 Median estimation for sound films produced in Hollywood, 1929 to 1931 inclusive

For the silent films, Method A produces a result that lies in the confidence interval of the true median only 4 times out of twenty (P = 0.20 [0.04, 0.37]). If we use this method we can expect our estimate to be outside the given confidence interval 80% of the time. Method B fares better for the silent films when σ = 0.9: out of twenty trials, the estimate was within the confidence interval for the true median on 13 occasions (P = 0.65 [0.43, 0.82]) – but this still means that it provides a poor estimate for approximately 1 in 3 films. When σ is the value derived by MLE, then the number of estimates that fall in the confidence interval of the true median is zero.

For the sound films, Method A provides a good estimate on 21 out of 30 occasions (P = 0.70 [0.52, 0.83]); and for Method B (σ = 0.9), the median is also well estimated 21 times. When σ is the value derived by MLE, then the number of estimates that fall in the confidence interval of the true median is 25 (P = 0.83 [0.66, 0.93]). These three methods when applied to the sample of sound films provide good estimates for the same film on 14 occasions, two methods provide good estimates on a total of twelve occasions (but it was not necessarily the same two for each of these twelve films), and on one occasion only a single method provides a good estimate. There are three films for which no method provides a good estimate. The different methods, then, provide different results for the same films.

The different methods proposed by Salt perform inconsistently across the two samples, and also produce different results when applied to the same sample. Overall, neither method provides a sound means of estimating the median shot lengths, and relying on median shot lengths estimated by these methods in the analysis will incorporate a large degree of error into the results as at least 17% of those estimates can be expected to lie outside the 95% confidence interval of the true median.

Summary

Salt has made a number of assertions about the appropriate methodology for the statistical analysis of film style. When this methodology is examined in detail, and these claims are subject to statistical hypothesis tests, they cannot be justified:

• The mean shot length is not a reliable statistic of film style. The median and the geometric mean are both more reliable measures of central tendency for shot length distributions that are positively skewed with outlying data points.
• There is no evidence that the majority of films have shot lengths that are lognormally distribution, let alone any evidence to support the claim that the lognormal distribution is a ‘ruling distribution’ of film style. Consequently, the use of the geometric mean as a measure of central tendency is less reliable than that of the median.
• There is no evidence to support the claim that the characteristic shape factor of the distribution of shot length in a motion picture is 0.9; while the claim that the shape factor will lie in the interval 0.7 to 0.9 produces inconsistent results across the samples examined here, with between a quarter and a half of the films outside this interval.
• The methods for estimating the median shot length from the mean shot length are inconsistent, and are not sufficiently reliable. Use of these methods to estimate the median from the mean shot length will introduce a large amount of error into a study.

The implications for film studies are depressing. The mean shot length has been used as statistic of film style for over thirty years in a number of publications by a number of prominent film scholars (e.g. Barry Salt, David Bordwell, Warren Buckland, Charles O’Brien, Yuri Tsivian, Colin Crisp, etc.). Unfortunately all this research is simply wrong, and as these studies have been further cited by other scholars this mistake has been multiplied. There is now a whole range of so-called ‘statistical analyses’ of film style out there, but none of it is, in fact, correct. The statistical analysis of film style can make a significant and positive contribution to our understanding of the cinema, but this first requires an understanding of statistics. Before the statistical analysis of film style can be good film studies, it must first be good statistics. Good statistics is the one thing we do not have at present. This problem goes back 35 years and the introduction of the mean shot length as a statistic of film style.

What is truly disheartening is that the mistakes made by film scholars in this area are elementary: in the UK, knowing when to use the mean and the median is GCSE statistics. The current specification for the AQA statistics syllabus clearly requires students – not university professors with Ph.D.s, but 14-16 year old school pupils – to understand the ‘advantages and disadvantages of each of the three measures of location [mean, median, mode] in a given situation,’ and to provide a ‘reasoned choice of a measure of location appropriate to the nature of the data and the purpose of the analysis.’ You can even get extra marks if you discuss the geometric mean! Any basic statistics course will tell you that you need to cite measures of dispersion alongside measures of central tendency. Every text book ever written on the subject discusses the meaning of the word ‘significant’ in the context of statistics. The application of statistics in film studies falls below these basic standards.

Do not take my word for it. Go and learn some statistics, or ask a statistician to show you how to do it. Get some data and do the analyses for yourself – and by analysis I mean actually formulate the hypotheses and do the tests rather than simply asserting two numbers are different and that this is ‘significant.’ Do not simply quote statistics when you do not know what from what population the sample was drawn, or when you do not know what the statistics are supposed to describe, or when you do not know what decision rule was employed, or when you do not know what tests were used. There really is nothing difficult about any of this.

Nulius addictus judicare in verbia magistri

Notes

1. I have assumed that film scholars will be using statistical tests to test hypotheses about data, but I have not actually come across anyone who has used a single statistical test in film studies. It is typical for film scholars to cite some means (without any accompanying measure of dispersion), and then simply to assert that a difference does or does not exist and that such a result is ‘significant.’ What they mean by ‘significant’ is not clear, but this is a term with a precise meaning in statistics and should not be abused. The statistical analysis of film style is scarcely statistics.
2. The reason for using the population variance is to be consistent with the MLE values given for σ, which were calculated based on the population standard deviation.
3. See Salt’s comment to my post ‘Testing Normality in Cinemetrics‘ dated 21 May 2009.
4. See Salt’s comment to my post ‘The impact of sound on film style‘ dated 25 September 2009. Note that in his comment to this post and the one cited above in note 3 Salt gives two different sets of figures for a set of 40 films I tested by the same method here, and that he gets both of them wrong. I can only assume that he has counted some films that appear in different posts twice. For the record (1) this is not a representative sample drawn from a population (2) there are 40 films in the table, and (3) half the films (20) have lognormal shot lengths. If we add the fifty films above to those forty films (remembering to remove the ones that overlap) we have a total set of 81 films of which 35 have lognormal shot lengths.
5. See Salt’s comment to my post ‘Location and spread in shot length distributions‘ dated 15 November 2009.
6. See Salt’s comment to my post ‘The impact of sound on film style‘ dated 25 September 2009.
7. A method for constructing exact confidence intervals for the median has been described by Bonnet and Price (2002), and there is a spreadsheet that can be downloaded to do this automatically. By all accounts this should be a better method than the binomial, but I have not been able to get hold of a copy of the article in which this method is described and so I am reluctant to use it without first understanding how it works.

References

Agresti A and Coull B 1998 Approximate is better than ‘exact’ for interval estimation of binomial proportions, The American Statistician 52: 119-126.

Bonett DG and Price RM 2002 Statistical inference for a linear function of medians: confidence intervals, hypothesis testing, and sample size requirements, Psychological Methods 7 (3): 370-383.

Curwin J and Slater R 2008 Quantitative Methods for Business Decisions, sixth edition. London: Thomson Learning.

Cotsaces C, Nikolaidis N, and Pitas I 2009 Semantic video fingerprinting and retrieval using face information, Signal Processing: Image Communication 24 (7): 598-613.

Fujita K 1989 Shot length distributions in educational TV programmes, Bulletin of the National Institute of Multimedia Education 2: 107-116.

Geyer CJ 2006 Breakdown point theory notes, http://www.stat.umn.edu/geyer/5601/notes/break.pdf, accessed 9 December 2009.

Looney SW and Gulledge TR 1985 Use of the correlation coefficient with normal probability plots, The American Statistician 39 (1): 75-79.

Olsson U 2005 Confidence intervals for the mean of a lognormal distribution, Journal of Statistics Education, Volume 13, Number 1, www.amstat.org/publications/jse/v13n1/olsson.html, accessed 18 November 2009.

Salt B 1974 Statistical style analysis of motion pictures, Film Quarterly 28 (1): 13-22.

Salt B 2006 Moving into Pictures: More on Film History, Style, and Analysis. Starwood, London.

Taskiran CM and Delp EJ 2002 A study on the distribution of shot lengths for video analysis, SPIE Conference on Storage and Retrieval for Media Databases, 20-25 January 2002, San Jose, CA. Available online: http://ctaskiran.com/papers/2002_ei_shotlen.pdf, accessed 7 August 2009.

Truong BT and Venkatesh S 2005 Finding the optimal temporal partitioning of video sequences, Proceedings of IEEE International Conference on Multimedia and Expo, 6-9 July 2005, Amsterdam, Netherlands: 1182-1185.

Vasconcelos N and Lippman A 2000 Statistical models of video structure for content analysis and characterization, IEEE Transactions on Image Processing 9 (1): 3-19.

Wessa P 2008 Maximum-likelihood lognormal distribution fitting (v1.0.2) in free statistics software (v1.1.23-r4), Office for Research Development and Education, http://www.wessa.net/rwasp_fitdistrlnorm.wasp/, accessed 15 November 2009.