Blog Archives

Neuroscience and the cinema

This week some papers on the neuroscience of viewing and remembering films.

Carvalho S, Leite J, Galdo-Álvarez S, and Gonçalves OF 2011 Psychophysiological correlates of sexually and non-sexually motivated attention to film clips in a workload task, PLoS One 6 (12): e29530.

Some authors have speculated that the cognitive component (P3) of the Event-Related Potential (ERP) can function as a psychophysiological measure of sexual interest. The aim of this study was to determine if the P3 ERP component in a workload task can be used as a specific and objective measure of sexual motivation by comparing the neurophysiologic response to stimuli of motivational relevance with different levels of valence and arousal. A total of 30 healthy volunteers watched different films clips with erotic, horror, social-positive and social-negative content, while answering an auditory oddball paradigm. Erotic film clips resulted in larger interference when compared to both the social-positive and auditory alone conditions. Horror film clips resulted in the highest levels of interference with smaller P3 amplitudes than erotic and also than social-positive, social-negative and auditory alone condition. No gender differences were found. Both horror and erotic film clips significantly decreased heart rate (HR) when compared to both social-positive and social-negative films. The erotic film clips significantly increased the skin conductance level (SCL) compared to the social-negative films. The horror film clips significantly increased the SCL compared to both social-positive and social-negative films. Both the highly arousing erotic and non-erotic (horror) movies produced the largest decrease in the P3 amplitude, a decrease in the HR and an increase in the SCL. These data support the notion that this workload task is very sensitive to the attentional resources allocated to the film clip, although they do not act as a specific index of sexual interest. Therefore, the use of this methodology seems to be of questionable utility as a specific measure of sexual interest or as an objective measure of the severity of Hypoactive Sexual Desire Disorder.

Frings L, Mader I, and Hüll M 2010 Watching TV news as a memory task – brain activation and age effects, BMC Neuroscience 11: 106.


Neuroimaging studies which investigate brain activity underlying declarative memory processes typically use artificial, unimodal laboratory stimuli. In contrast, we developed a paradigm which much more closely approximates real-life situations of information encoding.


In this study, we tested whether ecologically valid stimuli – clips of a TV news show – are apt to assess memory-related fMRI activation in healthy participants across a wide age range (22-70 years). We contrasted brain responses during natural stimulation (TV news video clips) with a control condition (scrambled versions of the same clips with reversed audio tracks). After scanning, free recall performance was assessed.


The memory task evoked robust activation of a left-lateralized network, including primarily lateral temporal cortex, frontal cortex, as well as the left hippocampus. Further analyses revealed that – when controlling for performance effects – older age was associated with greater activation of left temporal and right frontal cortex.


We demonstrate the feasibility of assessing brain activity underlying declarative memory using a natural stimulation paradigm with high ecological validity. The preliminary result of greater brain activation with increasing age might reflect an attempt to compensate for decreasing episodic memory capacity associated with aging.

Furman O, Dorfman N, Hasson U, Davachi L, and Dudai Y 2007 They saw a movie: long-term memory for an extended audiovisual narrative, Learning and Memory 14 (6): 457-467.

We measured long-term memory for a narrative film. During the study session, participants watched a 27-min movie episode, without instructions to remember it. During the test session, administered at a delay ranging from 3 h to 9 mo after the study session, long-term memory for the movie was probed using a computerized questionnaire that assessed cued recall, recognition, and metamemory of movie events sampled ∼20 sec apart. The performance of each group of participants was measured at a single time point only. The participants remembered many events in the movie even months after watching it. Analysis of performance, using multiple measures, indicates differences between recent (weeks) and remote (months) memory. While high-confidence recognition performance was a reliable index of memory throughout the measured time span, cued recall accuracy was higher for relatively recent information. Analysis of different content elements in the movie revealed differential memory performance profiles according to time since encoding. We also used the data to propose lower limits on the capacity of long-term memory. This experimental paradigm is useful not only for the analysis of behavioral performance that results from encoding episodes in a continuous real-life-like situation, but is also suitable for studying brain substrates and processes of real-life memory using functional brain imaging.

Hasson U, Furman O, Clark D, Dudai Y, and Davachi L 2008 Enhanced intersubject correlations during movie viewing correlate with successful episodic encoding, Neuron 57 (3): 452-462.

While much has been learned regarding the neural substrates supporting episodic encoding using highly controlled experimental protocols, relatively little is known regarding the neural bases of episodic encoding of real-world events. In an effort to examine this issue, we measured fMRI activity while observers viewed a novel TV sitcom. Three weeks later, subsequent memory (SM) for the narrative content of movie events was assessed. We analyzed the encoding data for intersubject correlations (ISC) based on subjects’ subsequent memory (ISC-SM) performance to identify brain regions whose BOLD response is significantly more correlated across subjects during portions of the movie that are successfully as compared to unsuccessfully encoded. These regions include the parahippocampal gyrus, superior temporal gyrus, anterior temporal poles, and the temporal-parietal junction. Further analyses reveal (1) that these correlated regions can display distinct activation profiles and (2) that the results seen with the ISC-SM analysis are complementary to more traditional linear models and allow analysis of complex time course data. Thus, the ISC-SM analysis extends traditional subsequent memory findings to a rich, dynamic and more ecologically valid situation.

Hasson U, Malach R, and Heeger DJ 2010 Reliability of cortical activity during natural stimulation, Trends in Cognitive Sciences 14 (1): 40-48.

Response reliability is complementary to more conventional measurements of response amplitudes, and can reveal phenomena that response amplitudes do not. Here we review studies that measured reliability of cortical activity within or between human subjects in response to naturalistic stimulation (e.g., free viewing of movies). Despite the seemingly uncontrolled nature of the task, some of these complex stimuli evoke highly reliable, selective, and time-locked activity in many brain areas, including some brain regions that often do not show much response modulation with conventional experimental protocols. This activity provides an opportunity to address novel questions concerning natural vision, temporal scale of processing, memory, and the neural basis of inter-group differences.

Jääskeläinen IP, Koskentalo K, Balk MH, Autti T, Kauramäki J, Pomren C, and Sams M 2008 Inter-Subject Synchronization of Prefrontal Cortex Hemodynamic Activity During Natural Viewing, The Open Neuroimaging Journal 2: 14-19.

Hemodynamic activity in occipital, temporal, and parietal cortical areas were recently shown to correlate across subjects during viewing of a 30-minute movie clip. However, most of the frontal cortex lacked between-subject correlations. Here we presented 12 healthy naïve volunteers with the first 72 minutes of a movie (“Crash”, 2005, Lions Gate Films) outside of the fMRI scanner to involve the subjects in the plot of the movie, followed by presentation of the last 36 minutes during fMRI scanning. We observed significant between-subjects correlation of fMRI activity in especially right hemisphere frontal cortical areas, in addition to the correlation of activity in temporal, occipital, and parietal areas. It is possible that this resulted from the subjects following the plot of the movie and being emotionally engaged in the movie during fMRI scanning. We further show that probabilistic independent component analysis (ICA) reveals meaningful activations in individual subjects during natural viewing.

And following on from this research:

Kauppi J-P, Jääskeläinen IP, Sams M, and Tohka J 2010 Inter-subject correlation of brain hemodynamic responses during watching a movie: localization in space and frequency, Frontiers in Neuroinformatics 4: 5.

Cinema is a promising naturalistic stimulus that enables, for instance, elicitation of robust emotions during functional magnetic resonance imaging (fMRI). Inter-subject correlation (ISC) has been used as a model-free analysis method to map the highly complex hemodynamic responses that are evoked during watching a movie. Here, we extended the ISC analysis to frequency domain using wavelet analysis combined with non-parametric permutation methods for making voxel-wise statistical inferences about frequency-band specific ISC. We applied these novel analysis methods to a dataset collected in our previous study where 12 subjects watched an emotionally engaging movie “Crash” during fMRI scanning. Our results suggest that several regions within the frontal and temporal lobes show ISC predominantly at low frequency bands, whereas visual cortical areas exhibit ISC also at higher frequencies. It is possible that these findings relate to recent observations of a cortical hierarchy of temporal receptive windows, or that the types of events processed in temporal and prefrontal cortical areas (e.g., social interactions) occur over longer time periods than the stimulus features processed in the visual areas. Software tools to perform frequency-specific ISC analysis, together with a visualization application, are available as open source Matlab code.

Wang HX, Freeman J, Merriam EP, Hasson U, and Heeger DJ 2012 Temporal eye movement strategies during naturalistic viewing, Journal of Vision 12 (1): 16.

The deployment of eye movements to complex spatiotemporal stimuli likely involves a variety of cognitive factors. However, eye movements to movies are surprisingly reliable both within and across observers. We exploited and manipulated that reliability to characterize observers’ temporal viewing strategies while they viewed naturalistic movies. Introducing cuts and scrambling the temporal order of the resulting clips systematically changed eye movement reliability. We developed a computational model that exhibited this behavior and provided an excellent fit to the measured eye movement reliability. The model assumed that observers searched for, found, and tracked a point of interest and that this process reset when there was a cut. The model did not require that eye movements depend on temporal context in any other way, and it managed to describe eye movements consistently across different observers and two movie sequences. Thus, we found no evidence for the integration of information over long time scales (greater than a second). The results are consistent with the idea that observers employ a simple tracking strategy even while viewing complex, engaging naturalistic stimuli.

Cognitive film theory: 2011 bibliographical update

Last January I post a list of papers from 2010 on the broadly defined topic of cognitive film theory, and this year I’m doing the same for the past twelve months. There is something for everyone here (check out Neal et al‘s paper on eating popcorn in the cinema), but I would particularly point you in the direction of a special issue of Science in Context on ‘Cinematography, Seriality, and the Sciences’ from last year, which you can find here.

As ever, this list is not exhaustive but it should be accurate. I’ve included some interesting sounding papers that can be accessed as e-publications ahead of print, and so the year of publication may change when they move from digital to analogue form.

  1. Bacon H 2011 The extent of mental completion of films, Projections 5 (1): 31-50.
  2. Berliner T and Cohen DJ 2011 The illusion of continuity: active perception and the classical editing system, Journal of Film and Video 63 (1): 44-63.
  3. Brown W2011 Resisting the psycho-logic of intensified continuity, Projections 5 (1): 69-86.
  4. Caputo NM and Rouner D 2011 Narrative processing of entertainment media and mental illness stigma, Health Communication 6 (7): 595-604. DOI: 10.1080/10410236.2011.560787.
  5. Cartwright L 2011 The hands of the projectionist, Science in Context 24 (3): 443-64. DOI: 10.1017/S0269889711000184.
  6. Cook RF 2011 Correspondences in visual imaging and spatial orientation in dreaming and film viewing, Dreaming 21 (2): 89-104. DOI: 10.1037/a0022866.
  7. Coyne SM, Nelson DA, Robinson SL, and Gundersen NC 2011 Is viewing ostracism on television distressing?, The Journal of Social Psychology 151 (3): 213-217. DOI: 10.1080/00224540903365570.
  8. Curtis S 2011 ‘Tangible as tissue:’ Arnold Gesell, infant behavior, and film analysis, Science in Context 24 (3): 417-442. DOI: 10.1017/S0269889711000172.
  9. Edelstein RS, Kean EL, and Chopik WJ 2011 Women with an avoidant attachment style show attenuated estradiol responses to emotionally intimate stimuli, Hormones and Behaviour, in press, DOI:10.1016/j.yhbeh.2011.11.007.
  10. Ghazanfar AA and Shepherd SV 2011 Monkeys at the movies: what evolutionary cinematics tells us about film, Projections 5 (2): 1-25.
  11. Haxby JV, Guntupalli JS, Connolly AC, Halchenko YO, Conroy BR, Gobbini MI, Hanke M, and Ramadge PJ 2011 A common, high-dimensional model of the representational space in human ventral temporal cortex, Neuron 72 (2): 404-16. DOI: 10.1016/j.neuron.2011.08.026.
  12. Hoeckner, B, Wyatt EW, Decety J and Nusbaum H 2011 Film music influences how viewers relate to movie characters, Psychology of Aesthetics, Creativity, and the Arts 5 (2): 146-153. DOI: 10.1037/a0021544
  13. Howarth PA 2011 Potential hazards of viewing 3-D stereoscopic television, cinema and computer games: a review, Ophthalmic and Physiological Optics 31 (2): 111-122. DOI: 10.1111/j.1475-1313.2011.00822.x.
  14. Jones S 2011 The pure moment of murder: the symbolic function of bodily interactions in horror films, Projections 5 (2): 96-114.
  15. Kano F and Tomonaga M 2011 Species difference in the timing of gaze movement between chimpanzees and humans, Animal Cognition 14 (6): 879-892. DOI: 10.1007/s10071-011-0422-5.
  16. Kovács AB 2011 Causal understanding and narration, Projections 5 (1): 51-68.
  17. Lavaur J-F and Bairstow D 2011 Languages on the screen: is film comprehension related to the viewers’ fluency level and to the language in the subtitles?, International Journal of Psychology, 46 (6): 455-462. DOI:10.1080/00207594.2011.565343.
  18. Lerner Y, Honey CJ, Silbert LJ, and Hasson U 2011 Topographic mapping of a hierarchy of temporal receptive windows using a narrated story, The Journal of Neuroscience 31 (8): 2906-2915. DOI:10.1523/JNEUROSCI.3684-10.2011.
  19. Magliano JP and Zacks JM 2011 The impact of continuity editing in narrative film on event segmentation, Cognitive Science 35 (8): 1489-1517. DOI: 10.1111/j.1551-6709.2011.01202.x
  20. Mital PK, Smith TJ, Hill R, and Henderson JM 2011 Clustering of gaze during dynamic scene viewing is predicted by motion, Cognitive Computation 3 (1): 5-24. DOI: 10.1007/s12559-010-9074-z.
  21. Neal DT, Wood W, Wu M, and Kurlander D 2011 The pull of the past: when do habits persist despite conflict with motives?, Personality and Social Psychology Bulletin 37 (11): 1428-1437. DOI: 10.1177/0146167211419863.
  22. Nishimoto S, Vu AT, Naselaris T, Benjamini Y, Yu B, and Gallant JL 2011 Reconstructing visual experiences from brain activity evoked by natural movies, Current Biology 21 (19): 1641-1646. DOI: 10.1016/j.cub.2011.08.031.
  23. Pannasch S, Selden DL, Velichkovsky BM, and Bridgeman B 2011 Apparent Phi-motion in sequences of Eisenstein’s October, Gestalt Theory 33 (1): 69-80.
  24. Pavlović I and Marković S 2011 The effect of music background on the emotional appraisal of film sequences, Psihologija 44 (1): 71-91. DOI: 10.2298/PSI1101071P.
  25. Plantinga C 2011 Folk psychology for film critics and scholars, Projections 5 (2): 26-50.
  26. Sabbadini A 2011 Cameras, mirrors, and the bridge space: a Winnicottian lens on cinema, Projections 5 (1): 17-30.
  27. Shibata H, Inui T, and Ogawa K 2011 Understanding interpersonal action coordination: an fMRI study, Experimental Brain Research 211 (3-4): 569-579. DOI: 10.1007/s00221-011-2648-5.
  28. Silvia PJ and Berg C 2011 Finding movies interesting: how appraisals and expertise influence the aesthetic experience of film, Empirical Studies of the Arts 29 (1): 73 – 88.
  29. Sonnenschein D 2011 Sound spheres: a model of psychoacoustic space in cinema, The New Soundtrack 1 (1): 13-27. DOI 10.3366/sound.2011.0003.
  30. Vandaele J 2011 What meets the eye: cognitive narratology for audio description, Perspectives: Studies in Translation, in press, DOI: 10.1080/0907676X.2011.632683.
  31. Wagner DD, Dal Cin S, Sargent JD, Kelley WM, and Heatherton TF 2011 Spontaneous action representation in smokers when watching movie characters smoke, The Journal of Neuroscience 31 (3): 894-898. DOI: 10.1523/​JNEUROSCI.5174-10.2011
  32. Wilson RT and Till BD 2011 Recall of preshow cinema advertising: a message processing perspective, Journal of Marketing Communications, in press, DOI: 10.1080/13527266.2010.538071.
  33. Zacks JM, Kurby CA, Eisenberg ML, and Haroutunian N 2011 Prediction error associated with the perceptual segmentation of naturalistic events, Journal of Cognitive Neuroscience 23 (12): 4057-4066. DOI: 10.1162/jocn_a_00078.
  34. Zumalde-Arregi I 2011 The filmic emotion: a comparative analysis of film theories, Revista Latina de Comunicación Social 66: 326-349. DOI: 10.4185/RLCS-66-2011-936-326-349-EN.

Pre-film studies research on film

A couple of weeks ago I posted about some empirical research on editing and the viewer’s experience of pace in the visual media from the 1970s.

A fascinating read on the same topic is the UNESCO report on mass communication research published in 1961, which presents a comprehensive and global bibliography of research on the influence of the cinema on children and adolescents. The report can be accessed here, and it is definitely worth taking an afternoon to read through it.

UNESCO 1961 The Influence of the Cinema on Children and Adolescents: An Annotated International Bibliography. Reports and Papers on Mass Communication 31. Paris: UNESCO.

The report provides details on 491 different pieces of research from around the world, and provides an insight into the type of research that was done before film studies came along. Areas covered include the social effects of cinema on young people, the use of film in education, film and juvenile delinquency, motives behind film choice, and there is good coverage of what we would now call cognitive film theory.

There are all sorts of interesting things to discover. For example, item 62 on the list provides a fascinating insight into the habits of younger viewers in Michigan in the 1940s.

Gibson, Harold J. (Mrs .) and Nahabedian, Vaskey (Mrs .) . A Survey of the Reading, Radio and Motion Picture Habits of Royal Oak Public School Students and their Parents. Royal Oak, Michigan, Royal Oak Public School, 1949, 21 p.

The average pupil in the school surveyed attends the cinema much more frequently than his parents. At the age of 8, he goes to the cinema once a week; until the age of 12 he attends the Saturday afternoon performance. When he reaches junior high school he goes to the cinema on Friday evening, generally with a friend. His parents help him in the selection of films, and he generally appreciates the films his parents consider suitable for him. Westerns, cartoons and animal films are his favourites; later his interest in westerns wanes and his interest in musicals grows. He now chooses films on the basis of cast and publicity. When he reaches high school, he will be more influenced in his choice by official film criticism, and he tends to have the same criteria as his parents.

Some of the research is a bit prosaic: item 61 is a study of the cinema-going habits of Italian young people and concludes that as they get older ‘boys go more frequently with girls.’ Isn’t that what the cinema is for?

However, I’m really not sure about the study from 1949 that showed Robert Flaherty’s Nanook of the North (1922) to some Italian 8-to-14 year olds (no. 130) that concluded that children had difficulty understanding the film. An 80 minute silent documentary about an Eskimo is hardly suitable viewing for children as young as eight. I know I’ve never been that enamoured of this film, but I’m certainly glad I didn’t have to watch it as a child. The comments about spatial awareness and recognition of regular geometric forms do sound more interesting though.

Albertini. Laura and Caruso, Ada, Percezione e interpretazione di imagini cinematografiche nei ragazzi. [Perception and interpretation of film images by children] In: Bianco e Nero, Rome, (X), 5 May 1949, p. 9-27. Also in: Baumgarten, Franziska, Compte rendu du lle Congrbs international de psychotechnique, Berne, 12-17 September 1949. La psychotechnique dans le monde moderne. Paris, Presses Universitaires de France, 1952, p. 557-561.

A study of the reactions of 576 children, aged 8 to 14, to Flaherty’s film “Nanook’. Four hundred and ninety children were questioned: 86 made unsolicited comments. Particularly apparent were the many errors in observation and the discrepancy between what actually occurred in the film and what the children thought they had seen. The rapid succession of images, the inability to understand clearly, to compare precisely and to interpret exactly when drawing up a report has the following results for children: real difficulties in making accurate comparisons as to sizes and likenesses, in recognizing regular geometric forms, in establishing the position of persons in relation to a known object, and in interpreting some of their movements and attitudes. Such difficulties as these do not seem to lessen proportionately as the child grows older. Further research is recommended to study the choice of motion-pictures for children of different age groups.

Perhaps the researchers might have asked the children if they wanted to watch Nanook?

There is an extensive series of entries describing quite detailed studies by the Japanese Ministry of Education on cinema attendance among young people from the 1930s that sound very interesting.

In his ‘Foreword’ to Moving Image Theory: Ecological Considerations, David Bordwell wrote that film studies ‘got off on the wrong foot methodologically. Instead of framing questions, to which competing theories might have responded in a common concern for enlightenment, film academics embraced a doctrine-driven conception of research’ (2005: xi, original emphasis). [Bordwell D 2005 Foreword, in JD Anderson and B Fisher Anderson (eds.) Moving Image Theory: Ecological Considerations. Carbondale: Southern Illinois University Press, 2005: ix-xii].

This is may be an accurate description of film studies, but it is not an accurate description of the study of film.

What stands out from reading much of the research in the UNESCO report  is that pre-film studies research in the cinema is (1) primarily concerned with psychology of the cinema and (2) that it is empirical research and is NOT doctrine-driven. And yet this research has had relatively little impact on film studies as it is taught in universities today. The institutionalisation of film studies as an academic discipline does not appear to have drawn on any of this tradition going back well into the silent era. Why not? And what are the consequences of this? What did these earlier researchers understand about the cinema that we have forgotten?

For example, Tim Smith has written about viewer’s eye movements when watching Hollywood films. You can find his blog describing his research here and his guest piece about eye movements in watching There Will Be Blood on David Bordwell’s blog is here. But if we go back to 1964 we can find this paper, which was addressing the same questions some 47 years ago.

Guba E, Wolf W, de Groot S, Knemeyer M, Van Atta R, and Light L 1964 Eye movements and TV viewing in children, Educational Technology Research and Development 12 (4): 386-401.

This paper, like those in the UNESCO report, does not feature in the film studies curriculum due to the collective amnesia of film scholars who, it would seem, simply ignored decades of prior research when creating university courses in film. Why this should be the case is one of the most important and most interesting questions in film studies.

In the comments on the last update to my bibliography on cognitive film theory, someone asked why I hadn’t included the French Filmology research of the 1940s and 1950s. You can find the bibliography and the comments here. Part of my response was that I simply did not come across this research that often and that translations of this work are relatively rare. It is much harder, for example, to find the works of Gilbert Cohen-Seat in English than it is to find those of Christian Metz. Why should this be so?

The study of film existed before film studies, and it existed as a body of empirical research that looked at how viewers experienced and comprehended the cinema, at the behaviour of audiences, and at the social impact of cinema. And it did so by asking questions years before Bordwell began writing about a mid-level research programme as a means of moving forward.

Film studies really screwed up the study of the cinema.

The Road Not Taken in Film Studies

Searching through the internet for something completely unrelated, I came across this piece of research that I think is worth sharing with the world.

Marks JN 1974 The Effects of Television Pacing Rates on Viewer Attitude and Interest Levels. Concordia University, Unpublished MA Thesis.


Three differently edited versions of the same TV presentation consisting of ‘rapid,’ ‘moderate,’ and ‘original’ pacing rates were shown to 120 grade 10 Ss to ascertain their differential effectiveness in changing viewer attitudes and maintainance of viewer interest. Ss were randomly divided into three TV viewing treatment groups and one non-viewing control group. Attitudes were measured by a 32 item post attitude questionnaire with reliability of m0.90. Interest was measured by a 20 item post interest questionnaire with reliability of 0.96. Single classification ANOVA and HSD Tests revealed significant differences (P < 0.05) on the attitude measure between the treatment groups and the control group and between the faster paced versions and the original version. the rapid paced version was found to be the most effective. While no significant difference were found in interest levels, the rapid pacing rate significantly increased viewer attention.

The thesis can be downloaded from the repository at Concordia University here.

It is worth reading this piece because it gives us an insight into early empirical  research of film and film viewership at a time when the hegemony of contemporary film theory was being established. 1974 was the year in which Screen published Colin McCabe’s ‘Realism and the cinema: notes on some Brechtian theses’ and Raymond Bellour’s ‘The obvious and the code;’ Jump Cut was founded; Film Theory and Criticism first appeared; and Christian Metz’s Film Language: A Semiotics of the Cinema and Language and Cinema were unleashed upon the world.

The thesis begins with a review of early research on film in an education context, and provides a series of references of research on the cinema before film studies. There are several references to research conducted on behalf of the US Army on the impact of the Why We Fight series. There are references to research going back to the early 1920s that are concerned with the nature of the viewer’s experience.

A couple of things stand out regarding this early work:

  • The concern with practical consequences for this early research – the apparent purpose behind much of this early research on educational film and television is to inform how future programmes should be produced to achieve their pedagogical aims
  • The concern with how people understand media texts – we would now consider this research to be a part of cognitive film theory as it is looks at the relationship between style (editing pace) and experience (attention and interest). The section on ‘Effects of Pacing on Cognitive and Affective Learning’ (from page 20) provides a summary of work in this area from the 1950s, but which would not be taken up in film studies until long after.

On page 6, Marks bemoans the fact there has been little research on style and its relationship to the viewer:

Although a great deal of film theory is based on the editing process, and has been since the beginning of motion picture production, there has been little to no systematic testing of the effects that the editing of films or television programs have on viewers.

It was not until the mid-1990s that film scholars began to address this lack as cognitivism began to become recognised as a central part of film theory. Marks sets out to address precisely this issue and asks a very simple question:

What relationship exists between the rate of pacing a highly visual information TV presentation and the attitude scores and interest levels of viewers?

This is carried out as a psychological research project, and clearly sets out the hypotheses being tested, the methodology being used, and the statistical analysis conducted. It therefore adopts a very different approach to the mainstream of film theory that had emerged in the examples given above.

This thesis is an example of how film studies could have been done – but instead it chose a different path.

Time Series Analysis of Top Hat (1935)

The editor Millie Moore (Johnny Got His Gun, Go Tell The Spartans), said

… one of the most important jobs of the picture editor is to control the tempo and pace of the story (Yewdall 2007: 156).

The ebb and flow of pace and tempo determines the dramatic form of a film, and it is through editing (along with camera motion and sound energy) that the viewer’s attention is structured. Dorai and Venkatesh (2001) observed that in Hollywood narrative cinema, large changes of pace occur at the boundaries of story segments (e.g. transitions between scenes), while smaller changes in pace are identified with local narrative events of high dramatic import. Similarly, Cutting et al. (2011) noted that within each quarter and possibly each act of a Hollywood film there is a pattern of general shortening and then lengthening of shots reflecting a fluctuating intensification of continuity. Different emotional states are associated with different editing styles (Kang 2002). In the television schedule, adverts are edited more quickly than the programmes around them in order to attract the viewer’s attention and to improve product recall (Young 2007).

It would seem natural that the methods of time series analysis could help us to describe the evolution of the tempo and pace over the course of a film and thereby to understand how and why this element of film style changes.

However, there are a number of problems:

  • Time is not an independent variable: typically we apply time series methods to understand how some variable (e.g. stock prices, animal populations, etc) changes as a function of time, but here the variable of interest is time itself (i.e. the amount of time between two edits). This does not make time series analysis impossible, but it does require careful interpretation of the results: for example, spectral analysis will be event-based rather than time-based, and will show the number of events per cycle rather than the duration of the cycle in some unit of time. Treating this data as a ‘standard’ time series may lead to incorrect interpretation of the style of a film.
  • Shot length data is typically positively skewed with a number of outliers: many common methods of time series analysis (e.g. running means, autocorrelation functions) assume that the data is normally distributed, but this is not the case for the shot lengths in a motion picture; and failing to take this into account can lead to flawed conclusions and erroneous estimations of parameters for time series models.
  • Shot length data may exhibit nonlinear characteristics: many time series methods assume that the data is linear, but we may find that the style of a film exhibits conditional heteroscedasticity (e.g. the variance of shot lengths in a rapidly edited action sequence will be lower than in slower dialogue sequences), that any cycles present may be asymmetric, or that there are abrupt changepoints in style as one scene ends and another begins. Other nonlinear features may also be apparent.

These problems can be overcome by using ordinal or rank-based methods that make fewer assumptions about the distribution of the data and allows us to conduct exploratory data analysis before deciding on how to model the evolution of style in a film. Crucially, we need not be concerned that time is the variable of interest as these methods require only that the data is ordered – which in this case means the order in which they occurred (shot 1 is the first shot, shot 2 is the second, …). Two methods are illustrated here: running Mann Whitney Z statistics and the order structure matrix. The data set used here is for Top Hat (1935), and can be accessed here as an Excel file: Nick Redfern – Top Hat.

The running Mann-Whitney Z statistic

The Mann-Whitney U test is a nonparamteric test of the null hypothesis that two random variables are stochastically equal. For an introduction to the Mann Whitney U Test see here. Steve Mauget (2003, 2011) has applied the Mann-Whitney U test to time series analysis of climate data by using moving windows to sample the ranks of shots in order to identify regimes of high and low ranking data points. This method can be used to identify trends in the time-ordered data, to identify any intermittent cyclical regimes, and to identify changepoints in the series as the style of a film evolves. This method is akin to using a moving average, but instead of looking at the level in successive windows, we are looking at the ranks of the data.

The first step in generating a time series is to rank the N shots in a film from the smallest to the largest, with tied values assigned the average of the ranks they would have been assigned if there were no ties: if x2 and x3 have the same value they are assigned an average rank of (2+3)/2 = 2.5. The ranks are then sampled using a window of size n1 , and the sum of the ranks of the shots in this window (R1) calculated. The values of n1 and R1 are used to calculate a U statistic by

and, if the sample is sufficiently large (n1 ≥ 10), then this can be transformed to a Z statistic by

If we plot the set of Z statistics produced by applying this method to Top Hat we get the time series in Figure 1, which was constructed using a sampling window of 20 shots.The significance of the Z statistic can be determined with reference to a standard normal distribution. Thus if α = 0.05, the critical z-value is ± 1.96; and so when Z ≥ 1.96 we will identify a significant cluster of high-ranking shots (i.e. long takes) and when Z ≤ 1.96 we will identify a significant cluster of low ranking shots (i.e. short takes).

The series in Figure 1 contains a lot of redundant information because consecutive windows overlap the same shots (i.e. if n1 = 20 then nineteen of the shots in window 1 will also appear in window 2), and so the windows we are interested in are the most-significant non-overlapping windows.

Figure 1 Running Mann-Whitney Z statistics for Top Hat (1935) using a 20 shot window , with significance at Z = ± 1.96

From Figure 1, we can see that Top Hat has a number of peaks and troughs corresponding to clusters of longer and shorter shots.

The first peak (A) includes the meeting between Jerry and Horace at the beginning of the film that sets up the story and the first musical number ‘No Strings (I’m Fancy Free).’ Jerry’s performance of this number is interrupted by Dale, whom he has woken with his dancing, and there is a sequence of a more rapidly edited shot-reverse shot pattern that occurs at the first trough (1). After Dale returns to her room, Jerry decides to cover the floor with sand and dances to a reprise of ‘No Strings,’ and this can be seen in the second peak (B). The following morning, Jerry takes the place of a Hansome cab driver and escorts Dale to the stables, and this is a second quickly edited dialogue scene that occurs at 2. The peak at C occurs with the second musical number, ‘Isn’t This a Lovely Day (to be Caught in the Rain).’ The troughs at 3 and 4 coincide with Dale mistaking Jerry for Horace in the hotel lobby, and a subsequent sequence which cross-cuts between Dale and Jerry in different hotel rooms after the former has slapped the latter. The peak Z statistic occurs at D, which is the sequence at the theatre in which Jerry and Horace talk in the dressing room before Jerry goes on stage to perform ‘Top Hat, White Tie, and Tails.’ This first half of the films takes place in London, and the peaks and troughs are associated with particular aspects of the musical comedy: the peaks (i.e. the clusters of higher ranked and – therefore longer – shots) are associated with the musical numbers, while the troughs are associated with the comedy story line of the mix up in the romance of Dale and Jerry.

As the action moves to Italy, we get trough at 5, which is the sequence in which Dale and Madge chat next to the canal, and 6, which is another sequence cross-cut between locations as Dale and Jerry speak on the phone. The peak at E occurs at the third musical number, ‘Cheek to Cheek,’ and its extended dance sequence. So far we have the same pattern that we saw in the London sequences: comedy is quick and musical slow. However, the peak at F is not associated with a musical number and spans the three scenes. This peak includes the end of the sequence in which Jerry and Horace talk (after Madge has given her husband a black eye), the long static takes in which Dale accepts Beddini’s proposal (as the melody for ‘Cheek to Cheek’ is played in the background), and the beginning of the next sequence as Jerry and Horace are asked by the hotel to vacate the bridal suite and they go onto to talk to Madge. This peak (F) is the only case of the narrative being characterised by a cluster of long shots in the film.

The next trough (7) is the sequence in which Jerry meets Beddini in the bedroom of the bridal suite before he takes Dale out on the canal. The trough at 8 is a cluster of short shots in which the narrative of mistaken identity is resolved between Beddini, Horace, and Madge (but not Dale and Jerry), and is followed by the final peak G, which is the last of the musical numbers, ‘The Piccolino,’ and the carnival dance sequence.

Overall, we can see from the editing structure revealed by using the running Mann-Whitney Z statistic that Top Hat is characterised by alternating clusters of longer and shorter takes, in which the former are typically associated with the musical parts of the film and the latter with the comedy-romance narrative.

This method can also be used to compare different films side by side, and in a few weeks I’ll post a paper using this method to analyse the time series of 15 BBC News bulletins that places this data into a single frame of reference so similarities and differences can be identified.

The order structure matrix

The same information we obtained from the running Mann-Whitney Z statistic can be seen in the order structure matrix for Top Hat in Figure 2, based on whether a shot is greater than or less than the shot that comes after it (Brandt 2005). To construct the matrix we assign a value of 1 when xs ≥ xt and a value of 0 when xs < xt. To make this easier to visualise we assign a colour to each value (1 = black, 0 = white) and plot the matrix in a grid. The dark patches in Figure 2 correspond to the peaks in Figure 1 and exhibit clustering of longer shots in the films, while the light patches correspond to the troughs of Figure 1 and show where the clusters of shorter shots are to be found. Although this plot looks complicated, once you get used to the method and are familiar with the events of the film you can simply read the changes in cutting style from left to right.

Figure 2 Order structure matrix for Top Hat (1935)

Figure 2 was produced by first calculating the matrix in Microsoft Excel; and then cutting and pasting the resulting array of 1s and 0s into the latest version of PAST (which you can download for free here), selecting the whole spreadsheet, and then choosing MATRIX from the PLOT menu.

Alternatively, you can produce Figure 2 by applying the filled.contour command in R to the matrix (see here for an explanation).

This method has a particular limitation: it is only really effective with large data sets, and it can be quite difficult to make out distinct patterns even when there are as many as 250 shots in a film. If, however, you have 500 or more shots, then it is an excellent place to start your exploration of the shot length data for a film.


Brandt C 2005 Ordinal time series analysis, Ecological Modelling 182: 229-238. [There is an online version of this paper that can be downloaded for free, but there is no URL associated with it. Search for the title and you’ll find it].

Cutting JE, Brunik KL, and DeLong JE 2011 How act structure sculpts shot lengths and shot transitions in Hollywood film, Projections 5 (1): 1-16.

Dorai C and Venkatesh S 2001 Bridging the semantic gap in content management systems: computational media aesthetics, in Proceedings 2001 International Conference on Computational Semiotics in Games and New Media. 10-12 September 2001, Amsterdam: 94-99.

Kang H-B 2002 Analysis of scene context related with emotional events, in Proceedings of the 10th ACM International Conference on Multimedia. 1-6 December 2002, Juan les Pins, France: 311-314.

Mauget SA 2003 Intra- to multidecadal climate variability over the continental United States: 1932–99, Journal of Climate 16: 3905–3916.

Mauget SA 2011 Time series analysis based on running Mann-Whitney Z statistics, Journal of Time Series Analysis 32 (1): 47–53.

Yewdall DL 2007 Practical Art of Motion Picture Sound, third edition. Burlington, MA: Focus Press.

Young C 2007 Fast editing speed and commercial performance, Admap 483: 30-33.

Cognitive Film Theory: Bibliographical Update

Back in January, when I posted an bibliographical update of works published on cognitive film theory, I was asked if I would do an update of the larger bibliography of research in this area that appeared the year before. Six months later, I have finally done this, and the link below will take you to the new version as a pdf file:

Nick Redfern – CognitiveFilmTheoryBibliography2-04

This bibliography now contains 425 different items, which means that 70 new references (highlighted in red) have been added since the old version was put on-line. By ‘new’ I mean only that items did not appear in the earlier version, and though many of these are from as recent as 2010 some are much older. Again, this bibliography makes no claim to being exhaustive, but what is there should be accurate. (I’ve also tidied up a few things since the last version but these were very minor).

I haven’t included any references from 2011 becasue the year isn’t over yet, and you’ll just have to wait until next January for that update.

One area that I have also left to one side, but which could make a very large contribution to the bibliography is research on cognition that uses film clips to explore cognition but which is not related to film studies (i.e. the research doesn’t necessarily use commerical films). One area of considerable research using film in this way are studies of ‘affective style,’ and a handful of references with links to papers by Richard J Davidson are listed below as examples. I may decide at a later date to add such research to the bibliography, but for the time being it remains in limbo.

Davidson RJ 1994 Asymmetric brain function, affective style, and psychopathology: the role of early experience and plasticity, Development and Psychology 6: 741-758.

Davidson RJ 1998 Anterior electrophysiological asymmetries, emotion, and depression: conceptual and methodological conundrums, Psychophyisiology 35: 607-614.

Davidson RJ 2003 Darwin and the neural bases of emotion and affective style, Annals of the New York Academy of Sciences 1000: 316-336.

Davidson RJ2004 Well-being and affective style: neural substrates and biobehavioural correlates, Philosophical Transactions of the Royal Society of London B: Biological Sciences 359 (1449): 1395-1411.

More Visual Illusions

I like visual illusions – though I must admit that the rotating snakes (Figure 1) from Professor Akiyoshi Kitaoka’a illusion pages makes me feel somewhat queasy.

Figure 1 Rotating snakes from Akiyoshi’s illusion pages (click on the image for a larger version or got to to see the illusion in all its glory).

You can find other versions of this illusion and many others at Akiyoshi’s illusion pages here, along with research papers that discuss the psychological basis of the illusions he features. An fMRI study of the above illusion is Kuriki I, Ashida H, Murakami I, and Kitaoka A 2008 Functional brain imaging of the Rotating Snakes illusion by fMRI, Journal of Vision 8 (10): 16, 1-10, and can be accessed here.

The Daily Cognition has twenty visual illusions here. presents a study of illusions from 1922 – Matthew Luckiesh’s Visual Illusions: Their Causes, Characteristics and Applications – in its entirety.

An interesting introduction to the role of visual illusions in psychological research is David Eagleman’s article on how the study of visual illusions has guided neuroscience research: Eagleman DM 2001 Visual illusions and neurobiology, Nature Reviews Neuroscience 2: 920-926.

Richard Gregory, who died in May of last year, conducetd a large amount of research on visual perception and illusions, and the website dedicated to his memory – and featuring some of his papers on illusion and perception – can be accessed here.

Another researcher in visual illusions is Cornelia Fermüller, and her website can be found here, and includes examples of illusions and her research on a computational theory of optical illusions in video sequences and stereo images. One paper worth reading is Ogale AS, Fermüller C, and Aloimonos Y 2005  Motion segmentation using occlusions, IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (6): 988-992.

We examine the key role of occlusions in finding independently moving objects instantaneously in a video obtained by a moving camera with a restricted field of view. In this problem, the image motion is caused by the combined effect of camera motion (egomotion), structure (depth), and the independent motion of scene entities. For a camera with a restricted field of view undergoing a small motion between frames, there exists in general a set of 3D camera motions compatible with the observed flow field even if only a small amount of noise is present, leading to ambiguous 3D motion estimates. If separable sets of solutions exist, motion-based clustering can detect one category of moving objects. Even if a single inseparable set of solutions is found, we show that occlusion information can be used to find ordinal depth, which is critical in identifying a new class of moving objects. In order to find ordinal depth, occlusions must not only be known, but they must also be filled (grouped) with optical flow from neighboring regions. We present a novel algorithm for filling occlusions and deducing ordinal depth under general circumstances. Finally, we describe another category of moving objects which is detected using cardinal comparisons between structure from motion and structure estimates from another source (e.g., stereo).

This paper from Alex Holcombe looks at the illusion of motion perception in the cinema:

Holcombe AO 2009 Seeing slow and seeing fast: two limits on perception, Trends in Cognitive Neuroscience 13 (5): 216-221. [This paper contains links to movies as part of the paper’s supplementary materials].

Video cameras have a single temporal limit set by the frame rate. The human visual system has multiple temporal limits set by its various constituent mechanisms. These limits seem to form two groups. A fast group comprises specialized mechanisms for extracting perceptual qualities such as motion direction, depth and edges. The second group, with coarse temporal resolution, includes judgments of the pairing of color and motion, the joint identification of arbitrary spatially separated features, the recognition of words and high-level motion. These temporally coarse percepts might all be mediated by high-level processes. Working at very different timescales, the two groups of mechanisms collaborate to create our unified visual experience.

Mel Slater’s paper looks at why we experience a sense of immersion in artificially created environments.

Slater M 2009 Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments, Philosophical Transactions of the Royal Society B 364 (1535): 3549-3557.

In this paper, I address the question as to why participants tend to respond realistically to situations and events portrayed within an immersive virtual reality system. The idea is put forward, based on the experience of a large number of experimental studies, that there are two orthogonal components that contribute to this realistic response. The first is ‘being there’, often called ‘presence’, the qualia of having a sensation of being in a real place. We call this place illusion (PI). Second, plausibility illusion (Psi) refers to the illusion that the scenario being depicted is actually occurring. In the case of both PI and Psi the participant knows for sure that they are not ‘there’ and that the events are not occurring. PI is constrained by the sensorimotor contingencies afforded by the virtual reality system. Psi is determined by the extent to which the system can produce events that directly relate to the participant, the overall credibility of the scenario being depicted in comparison with expectations. We argue that when both PI and Psi occur, participants will respond realistically to the virtual reality.

A different approach to the nature of immersion in visual perception of animated images can be found in this paper from Kenny Chow and Fox Harrell:

Chow KKN and Harrell DF 2009 Material-based imagination: embodied cognition in animated images, Cognition and Creativity, Digital Arts and Culture 2009, Arts Computation Engineering, UC Irvine, California,;jsessionid=A526BE7A0733A59BB004CD0AC05A9EB6.

Drawing upon cognitive science theories of conceptual blending and material anchors, as well as recent neuroscience results regarding mirror neurons, we argue that animated visual graphics, as embodied images whose understanding relies on our perceptual and motor apparati, connect both material and mental notions of images. Animated visual images mobilize a reflective process in which material-based imaginative construction and elaboration can take place. We call this process as “material-based imagination,” in contrast to the general notion of imagination as purely a mental activity. This kind of imagination is pervasive in today’s digitally mediated environments. By analyzing a range of digital artifacts from computer interfaces to digital artworks, we show the important role of imaginative blends of concepts in making multiple levels of meaning, including visceral sensation and metaphorical narrative imagining, to exemplify expressiveness and functionality. The implications of these analyses collectively form a step toward an embodied cognition approach to animation phenomena and toward recentralizing understanding of artistic and humanistic production in cognitive research.

Finally, the 2011 finalists for the Illusion of the Year contest can be found here.

The geographies of cinema II

This week some more links looking at geographies of film. The first post in this series with lots of other links, is here.

(As ever, the version of a paper linked to may not be the final published version).

But before we get to the papers, it is worth taking some time to visit Aether: The Journal of Media Geography, an open-access journal edited by Jim Craine, Jason Dittmer, and Chris Lukinbeal.

Aether offers a forum that examines the geography of media, including cinema, television, the Internet, music, art, advertising, newspapers and magazines, video and animation. It is our goal to provide a space for contributions to current issues surrounding these media, beginning with constructions of space & place, cultural landscapes, society, and identity.

The editorial from issue 1 can be accessed here: Lukinbeal C, Craine J, and Dittmer J 2007 Aether: A Prospectus, Aether: The Journal of Media Geography 1: 1-3.

Alderman DH and Popke EJ 2002 Humor and film in the geography classroom: learning from Michael Moore’s TV Nation, Journal of Geography 101: 228-239.

How can teachers use humour and film to convert geography classrooms into public spaces for thinking and talking about the world in a critical way? One useful resource for raising student consciousness and critical discussion is TV Nation-a satirical television newsmagazine show created, produced, and hosted by rebel filmmaker Michael Moore in the mid 1990s. TV Nation not only serves as a potential instructional resource for geographers but also provides teacher and student a springboard for re-thinking humor and television news as analytical/educational objects. Moore challenges the popular notion that humor should not be taken seriously. By combining laughter with harsh reality, he questions the legitimacy of established ways of seeing the world and provides a unique way of discussing the socially constructed and contested nature of space and place. TV Nation also challenges the value traditionally placed on claims of neutrality and objectivity in conventional television news narratives. By making his own perspectives clearly known, Moore exposes the positionality inherent in all media representations of place. Included in this paper is an annotated list of TV Nation segments available on video and a description of how one of these news segments was used in a college-level classroom to teach about the complexities and contradictions of free trade and globalization.

Aitken SC and Dixon DP 2006 Imagining geographies of film, Erdkunde 60 (4): 326-336.

To the extent that the geographic study of film has come of age, it is important to not only tie it to disciplinary issues but also to push theoretical boundaries. Geographic concern is often lacking a critical perspective, focusing primarily on the geographic realism of films rather than how they produce meaning. Geographers needed to elaborate insights through critical spatial theories, so that our studies are not only about filmic representations of space but are also about the material conditions of lived experience and everyday social practices. With this essay, we argue for more critical film geographies. In doing so, we note how a series of traditional and emergent geographic ‘primitives’ – landscapes, spaces/spatialities, mobilities, scales and networks – are reappraised and push disciplinary boundaries for geography and film studies in general.

Chanan M 1997 The changing geography of third cinema, Screen 38 (4): 372-388.

The next paper is not strictly relevant to film studies – it belongs firmly in the culture wars of the 1990s; but it adopts such an odd way of arguing about the relationship between science and postmodernism (via Mary Poppins as an allegory) that I thought it worth including.

Dixon DP and Jones JP 1996 For a supercalifragilisticexpialidocious scientific geography, Annals of the Association of American Geographers 86 (4): 767-779.

Contemporary geographic thought finds scientific approaches triangulated by critiques launched by various political economy, feminist, and poststructuralist positions. In aiming their conceptual arsenal at fixed understandings of scientific geography, however, such critiques run the danger of essentializing their intended target. Moreover, in the consequent stabilization of the trajectories taken by these critiques, the process of criticism itself becomes an unreflexive exercise. In this paper we deploy the resources of poststructuralism to achieve an antiessentialist reading of scientific geography that moves beyond mere repudiation and seeks instead to identify a redemptive moment within this constellation of ideas and practices. To do so, we draw upon a modern-day parable–Mary Poppins–whose film version we read as offering a panoramic on theoretical divisions in geography. Though ostensibly a story about an all-too-perfect nanny, the film’s key protagonists serve as allegorical figures animating our analysis. Fortunately for all concerned, the banker/patriarch comes to the realization that he too can countermand rather than reproduce the fixed spaces of everyday life.

Engert S and Spencer A 2009 International relations at the movies: teaching and learning about international politics through film, Perspectives 17 (1): 83-103.

For mainstream Political Science, ‘popular culture’ is still not considered worthy of serious investigation. Similarly, the idea of using movies as a pedagogical tool has remained at the margins. Nevertheless, film can be a valuable means of teaching university students about politics and international politics in particular. This paper identifies four distinct ways of using movies as a teaching tool: the first approach uses film to portray historical events such as the Cold War, and the second utilizes film to debate specific issues in international politics such as terrorism or genocide. The third approach examines movies as cultural narratives– e.g. anti-Americanism in Turkey –, while the fourth uses film to explain and criticize IR theories (here, for example, Post-Modernism is discussed with the help of the movie Pulp Fiction). The article examines the strengths and weaknesses of using film in the IR classroom in general and illustrates each of the four approaches by using examples from movies.

Kennedy C and Lukinbeal C 1997 Towards a holistic approach to geographic research on film, Progress in Human Geography 21 (1): 33-50.

Geographers’ interest in film has increased during the last 20 years. Methodological and theoretical perspectives tend, however, to be bipolar and reflect either cognitive or social approaches. Work reflecting these approaches is reviewed with geographic research grounded in transactionalism and postmodernism as examples. A geographic view of film that recognizes the importance of more than one theoretical framework, positions the cognitive and social in a continuum reaching from the individual to the societal, and makes traditional notions of scale antiquated is recommended. Research by geographers contesting the assumed objectivity in documentaries is reviewed as are geographers’ contributions to understanding the construction of meaning of urban and natural settings in films. Suggestions for future directions in film research are made.

Schlottman A and Miggelbrink J 2009 Visual geographies – an editorial, Social Geography 4 (1): 1-11.

This paper is about the relationship between the visual and geography in lots of different ways and does not focus specifically on film, but its concerns are relevant to geographies of cinema and its list of references is useful.

Weissbrod R 2008 Israeli literature and cinema in a web of intercultural relations: the reconciliation of conflicts on screen, Borderlands 7 (1):

One way of establishing intercultural contacts is to produce a cinematic adaptation of a literary work originating in another country. The present article examines three adaptations in which Israeli culture is involved: Lost Lover, directed by Roberto Faenza (1999), which is based on Avraham B. Yehoshua’s The Lover (1977); The Island on Bird Street, directed by Søren Kragh-Jacobsen (1997), which is based on a novel by Uri Orlev (1981); and Saint Clara, directed by Ari Folman and Ori Sivan (1996), which is based on The Ideas of Saint Clara (1986 [1981]) by the Czechs Jelena Masínová and Pavel Kohout. Films usually re-shape their literary sources, seeking to adapt them to new circumstances and to a new audience. A significant modification of the source is likely to take place when the transfer from literature to cinema is also an intercultural one. In the era of trans-national media, account should also be taken of the possibility that the filmmakers might endeavor to make the film universally acceptable rather than adapting it to a specific target culture. Against this background, the article examines how the films under consideration depoliticize (in the terminology of Barthes) historical, ideological and political issues referred to in the novels, replacing controversial stands with widely accepted values such as peace and love, probably in order to increase their appeal to diverse audiences. This applies especially to the treatment of Zionist ideology and history which the films prefer to marginalize or evade rather than criticize or endorse.

Finally, Cinema City was a course aimed at postgraduates run by Majilis, Max Mueller, and SNDT Women’s University in October 2010. The website for the course is here, and by going to ‘course infromation’ and then clicking on ‘resource materials,’ you can access lots of online papers on the cinema in urban space, Wim Wenders, spectatorial rights, Hindi cinema, and many other things.

Empirical studies in film style IV

It has been a while since I listed some research on the empirical analysis of film style – I could have sworn I did a post on this just before christmas, but apparently not.

First, a couple of general papers that outline the principles of video content analysis (VCA) and the research that has been done in this area. This piece (here) is a set of power point slides by Alan Hanjalic (see below), in which he summarises the goals of VCA, its applications, and the different approaches that have been adopted by researchers. A literature survey of work in VCA is given in the following paper:

Brezeale D and Cook DJ 2008 Automatic video classification : a survey of the literature, IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 38 (3): 416-430. DOI: 10.1109/TSMCC.2008.919173.

Earlier posts on the empirical analysis of film style can be accessed here, here, and here.

The papers referred to below all cover the relationship between emotion, style, and video content.

Arifin S and Chueng PYK 2006 User attention based arousal content modelling, IEEE International Conference on Image Processing, 8 November 2006, Atlanta, Georgia, USA.


The affective content of a video is defined as the expected amount and type of emotion that are contained in a video. Utilizing this affective content will extend the current scope of application possibilities. The dimensional approach to representing emotion can play an important role in the development of an affective video content analyzer. The three basic affect dimensions are defined as valence, arousal and control. This paper presents a novel FPGA-based system for modeling the arousal content of a video based on user saliency and film grammar. The design is implemented on a Xilinx Virtex-II xc2v6000 on board a RC300 board.

The poster for this paper can be accessed here.

Hanjalic A 2006 Extracting moods from pictures and sounds, IEEE Signal Processing Magazine 23 (2): 90-100. DOI: 10.1109/MSP.2006.1621452.

From the introduction:

Intensive research efforts in the field of multimedia content analysis in the past 15 years have resulted in an abundance of theoretical and algorithmic solutions for extracting the content-related information from audiovisual signals. The solutions proposed so far cover an enormous application scope and aim at enabling us to easily access the events, people, objects, and scenes captured by the camera, to quickly retrieve our favorite themes from a large music video archive (e.g., a pop/rock concert database), or to efficiently generate comprehensive overviews, summaries, and abstracts of movies, sports TV broadcasts, surveillance, meeting recordings, and educational video material. However, what about the task of finding exciting parts of a sports TV broadcast or funny and romantic excerpts from a movie? What about locating unpleasant video clips we would be reluctant to let our children watch? This article considers how we feel about the content we see or hear. As opposed to the cognitive content information composed of the facts about the genre, temporal content structure (shots, scenes) and spatiotemporal content elements (objects, persons, events, topics) we are interested in obtaining the information about the feelings, emotions, and moods evoked by a speech, audio, or video clip. We refer to the latter as the affective content, and to the terms such as “happy ” or “exciting ” as the affective labels of an audiovisual signal.

Hanjalic A and Xu L 2005 Affective video content and representation modelling, IEEE Transactions on Multimedia 7 (1): 143-154. DOI: 10.1109/TMM.2004.840618.


This paper looks into a new direction in video content analysis – the representation and modelling of affective video content. The affective content of a given video clip can be defined as the intensity and type of feeling or emotion (both are referred to as affect) that are expected to arise in the user while watching that clip. The availability of methodologies for automatically extracting this type of video content will extend the current scope of possibilities for video indexing and retrieval. For instance, we will be able to search for the funniest or the most thrilling parts of a movie, or the most exciting events of a sport program. Furthermore, as the user may want to select a movie not only based on its genre, cast, director and story content, but also on its prevailing mood, the affective content analysis is also likely to contribute to enhancing the quality of personalizing the video delivery to the user. We propose in this paper a computational framework for affective video content representation and modelling. This framework is based on the dimensional approach to affect that is known from the field of psychophysiology. According to this approach, the affective video content can be represented as a set of points in the two-dimensional (2-D) emotion space that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect).We map the affective video content onto the 2-D emotion space by using the models that link the arousal and valence dimensions to low-level features extracted from video data. This results in the arousal and valence time curves that, either considered separately or combined into the so-called affect curve, are introduced as reliable representations of expected transitions from one feeling to another along a video, as perceived by a viewer.

Machajdik J and Hanbury A 2010 Affective image classification using features inspired by psychology and art theory, ACM Multimedia Conference 25-29 October 2010, Firenze, Italy.


Images can affect people on an emotional level. Since the emotions that arise in the viewer of an image are highly subjective, they are rarely indexed. However there are situations when it would be helpful if images could be retrieved based on their emotional content. We investigate and develop methods to extract and combine low-level features that represent the emotional content of an image, and use these for image emotion classification. Specifically, we exploit theoretical and empirical concepts from psychology and art theory to extract image features that are specific to the domain of artworks with emotional expression. For testing and training, we use three data sets: the International Affective Picture System (IAPS); a set of artistic photography from a photo sharing site (to investigate whether the conscious use of colors and textures displayed by the artists improves the classification); and a set of peer rated abstract paintings to investigate the influence of the features and ratings on pictures without contextual content. Improved classification results are obtained on the International Affective Picture System (IAPS), compared to state of the art work.

This paper does not relate specifically to film, but I include it anyway becuase it is interesting to read alongside the other papers listed here and in the context of cognitive film theory. The pdf linked to for this paper is over 10MB, so it may be quite slow to download.

Soleymani M, Chanel G, Kierkels JJK, and Pun T 2008 Affective characterization of movie scenes based on multimedia content analysis and user’s physiological emotional responses, IEEE International Symposium on Multimedia, 15-17 December 2008, Berkeley, California, USA [Abstract only].


In this paper, we propose an approach for affective representation of movie scenes based on the emotions that are actually felt by spectators. Such a representation can be used for characterizing the emotional content of video clips for e.g. affective video indexing and retrieval, neuromarketing studies, etc. A dataset of 64 different scenes from eight movies was shown to eight participants. While watching these clips, their physiological responses were recorded. The participants were also asked to self-assess their felt emotional arousal and valence for each scene. In addition, content-based audio- and video-based features were extracted from the movie scenes in order to characterize each one. Degrees of arousal and valence were estimated by a linear combination of features from physiological signals, as well as by a linear combination of content-based features. We showed that a significant correlation exists between arousal/valence provided by the spectator’s self-assessments, and affective grades obtained automatically from either physiological responses or from audio-video features. This demonstrates the ability of using multimedia features and physiological responses to predict the expected affect of the user in response to the emotional video content.

Yoo HW and Cho SB 2007 Video scene retrieval with interactive genetic algorithm, Multimedia Tools and Applications 34 (3): 317-336. DOI: 10.1007/s11042-007-0109-8.


This paper proposes a video scene retrieval algorithm based on emotion. First, abrupt/gradual shot boundaries are detected in the video clip of representing a specific story. Then, five video features such as “average colour histogram,” “average brightness,” “average edge histogram,” “average shot duration,” and “gradual change rate” are extracted from each of the videos, and mapping through an interactive genetic algorithm is conducted between these features and the emotional space that a user has in mind. After the proposed algorithm selects the videos that contain the corresponding emotion from the initial population of videos, the feature vectors from them are regarded as chromosomes, and a genetic crossover is applied to those feature vectors. Next, new chromosomes after crossover and feature vectors in the database videos are compared based on a similarity function to obtain the most similar videos as solutions of the next generation. By iterating this process, a new population of videos that a user has in mind are retrieved. In order to show the validity of the proposed method, six example categories of “action,” “excitement,” “suspense,” “quietness,” “relaxation,” and “happiness” are used as emotions for experiments. This method of retrieval shows 70% of effectiveness on the average over 300 commercial videos.

Finally, a report from a couple of years ago that appeared in IEEE Spectrum about a jacket that lets you “feel the movies” to add a sense of touch to the emotional events in a film.

The jacket contains 64 independently controlled actuators distributed across the arms and torso. The actuators are arrayed in 16 groups of four and linked along a serial bus; each group shares a microprocessor. The actuators draw so little current that the jacket could operate for an hour on its two AA batteries even if the system was continuously driving 20 of the motors simultaneously.

So what can the jacket make you feel? Can it cause a viewer to feel a blow to the ribs as he watches Bruce Lee take on a dozen thugs? No, says Lemmens. Although the garment can simulate outside forces, translating kicks and punches is not what the actuators are meant to do. The aim, he says, is investigating emotional immersion.

The article can be accessed here.

Cognitive film theory: 2010 bibliography update

Last January, I posted my bibliography of books, articles, etc., on the subject of cogntive film theory and which can be accessed here. This post includes articles published in this area during 2010. The list is not exhaustive, but it is accurate.

  1. Addis, Michael, and Morris B. Holbrook, ‘Consumers’ Identification and Beyond: Attraction, Reverence, and Escapism in the Evaluation of Films,’ Psychology and Marketing 27 (9) 2010: 821–845. doi: 10.1002/mar.20359.
  2. Bartsch, Anne, ‘Vivid Abstractions: On the Role of Emotion Metaphors in Film Viewers’ Search for Deeper Insight and Meaning,’ Midwest Studies in Philosophy 34 (1) 2010: 240-260. doi: 10.1111/j.1475-4975.2010.00198.x.
  3. Bartsch, Anne, Markus Appel, and Dennis Storch, ‘Predicting Emotions and Meta-Emotions at the Movies: The Role of the Need for Affect in Audiences’ Experience of Horror and Drama,’ Communication Research 37 (2) 2010: 167-190. doi: 10.1177/0093650209356441.
  4. Blumstein, Daniel T., Richard Davitan, and Peter D. Kaye, ‘Do Film Soundtracks Contain Nonlinear Analogues to Influence Emotion?,’ Biology Letters 6 (6) 2010: 751-754. doi: 10.1098/rsbl.2010.0333.
  5. Bordwell, David, ‘The Part-time Cognitivist: A View from Film Studies,’ Projections 4 (2) 2010: 1-18. doi: 10.3167/proj.2010.040202.
  6. Branigan, Edward, ‘Soundtracks in Mind,’ Projections 4 (1) 2010: 41-67. doi: 10.3167/proj.2010.040104.
  7. Bruun Vaage, Margrethe, ‘Fiction Film and the Varieties of Empathic Engagement,’ Midwest Studies in Philosophy 34 (1) 2010: 158-79. doi: 10.1111/j.1475-4975.2010.00200.x.
  8. Carroll, Noël, ‘Movies, the Moral Emotions, and Sympathy,’ Midwest Studies in Philosophy 34 (1) 2010: 1-19. doi: 10.1111/j.1475-4975.2010.00197.x.
  9. Currie, Gregory, ‘Bergman and the Film Image,’ Midwest Studies in Philosophy 34 (1) 2010: 323-339. doi: 10.1111/j.1475-4975.2010.00199.x.
  10. Cutting, James E., Jordan E. DeLong, and Christine E. Noether, ‘Attention and the Evolution of Hollywood Film,’ Psychological Science 21 (3) 2010: 432-439. doi: 10.1177/0956797610361679.
  11. Dadlez, E.M., ‘Seeing and Imagination: Emotional Response to Fictional Film,’ Midwest Studies in Philosophy 34 (1) 2010: 120-135. doi: 10.1111/j.1475-4975.2010.00194.x.
  12. Dennis, Tracy A., and Beylul Solomon, ‘Frontal EEG and Emotion Regulation: Electrocortical Activity in Response to Emotional Film Clips is Associated with Reduced Mood Induction and Attention Interference Effects,’ Biological Psychology 85 (3) 2010: 456-464. doi: 10.1016/j.biopsycho.2010.09.008.
  13. Deutsch, Stephan, ‘Psycho and the Orchestration of Anxiety,’ The Soundtrack 3 (1) 2010: 53-66. doi: 10.1386/st.3.1.53_1.
  14. Eder, Jens, ‘Understanding Characters,’ Projections 4 (1) 2010: 16-40. doi: 10.3167/proj.2010.040103
  15. Elliot, Paul, ‘The Eye, The Brain, The Screen: What Neuroscience Can Teach Film Theory,’ Excursions 1 (1) 2010: 1-16.
  16. Feagin, Susan L., ‘Film Appreciation and Moral Insensitivity,’ Midwest Studies in Philosophy 34 (1) 2010: 20-33. doi: 10.1111/j.1475-4975.2010.00197.x.
  17. Friend, Stacie, ‘Getting Carried Away: Evaluating the Emotional Influence of Fiction Film,’ Midwest Studies in Philosophy 34 (1) 2010: 77-105. doi:  10.1111/j.1475-4975.2010.00196.x.
  18. Gaut, Berys, ‘Empathy and Identification in Film,’ Midwest Studies in Philosophy 34 (1) 2010: 136–157. doi: 10.1111/j.1475-4975.2010.00211.x.
  19. Giovannelli, Alessandro, ‘Cognitive Value and Imaginative Identification: The Case of Kubrick’s Eyes Wide Shut,’ The Journal of Aesthetics and Art Criticism 68 (4) 2010: 355–366. doi: 10.1111/j.1540-6245.2010.01430.x.
  20. Grau, Christophe, ‘American History X, Cinematic Manipulation, and Moral Conversion,’ Midwest Studies in Philosophy 34 (1) 2010: 52-76. doi: 10.1111/j.1475-4975.2010.00210.x.
  21. Grodal, Torben, ‘High on Crime Fiction and Detection,’ Projections 4 (2) 2010: 64-85. doi: 10.3167/proj.2010.040205.
  22. Harold, James, ‘Mixed Feelings: Conflicts in Emotional Responses to Film,’ Midwest Studies in Philosophy 34 (1) 2010: 280-294. doi:  10.1111/j.1475-4975.2010.00209.x.
  23. Hirose, Yoriko, Alan Kennedy, and Benjamin W. Tatler, ‘Perception and Memory Across Viewpoint Changes in Moving Images,’ Journal of Vision 10 (4) 2010: 2. doi:10.1167/10.4.2.
  24. Hopkins, Robert, ‘Moving because Pictures? Illusion and the Emotional Power of Film,’ Midwest Studies in Philosophy 34 (1) 2010: 200-218. doi: 10.1111/j.1475-4975.2010.00208.x.
  25. Kauppi, Jukka-Pekka, Iiro P. Jääskeläinen, Mikko Sams, and Jussi Tohka, ‘Inter-subject Correlation of Brain Hemodynamic Responses During Watching a Movie: Localization in Space and Frequency,’ Frontiers in Neuroinformatics 4 2010: 5. doi: 10.3389/fninf.2010.00005.
  26. Kim, Seahwa, ‘The Rationality of Emotion toward Fiction,’ Midwest Studies in Philosophy 34 (1) 2010: 106-119. doi: 10.1111/j.1475-4975.2010.00203.x.
  27. Kim, Sue J. ‘Anger, Cognition, Ideology: What Crash Can Show Us about Emotion,’ Image and Narrative 11 (2) 2010:
  28. Laine, Tarva, ‘The Diving Bell and the Butterfly as an Emotional Event,’ Midwest Studies in Philosophy 34 (1) 2010: 295-305. doi: 10.1111/j.1475-4975.2010.00207.x.
  29. Lamarque, Peter, and Peter Goldie, ‘Whimsicality in the Films of Eric Rohmer,’ Midwest Studies in Philosophy 34 (1) 2010: 306-322. doi: 10.1111/j.1475-4975.2010.00206.x.
  30. Livingstone, Paisley, ‘On the Appreciation of Cinematic Adaptations,’ Projections 4 (2) 2010: 104-127. doi: 10.3167/proj.2010.040207.
  31. Matravers, Derek, ‘Why We Should Give Up on the Imagination,’ Midwest Studies in Philosophy 34 (1) 2010: 190-199. doi: 10.1111/j.1475-4975.2010.00205.x.
  32. Michel, Eva, ‘The Role of Individual Differences in Cognitive Skills in Children’s Learning Through Film,’ Journal of Media Psychology 22 (3) 2010: 105-113. doi: 10.1027/1864-1105/a000015.
  33. Nixon, Lizze, ‘I Focalize, You Focalize, We All Focalize Together: Audience Participation in Persepolis,’ Image and Narrative 11 (2) 2010:
  34. Oliver, Mary Beth, and Tilo Hartmann, ‘Exploring the Role of Meaningful Experiences in Users’ Appreciation of “Good Movies,”’ Projections 4 (2) 2010: 128-150. doi: 10.3167/proj.2010.040208.
  35. Planitnga, Carl, ‘“I Followed the Rules, and They All Loved You More”: Moral Judgment and Attitudes toward Fictional Characters in Film,’ Midwest Studies in Philosophy 34 (1) 2010: 34-51. doi: 10.1111/j.1475-4975.2010.00204.x.
  36. Planitnga, Carl, ‘Affective Incongruity and The Thin Red Line,’ Projections 4 (2) 2010: 86-103. doi: 10.3167/proj.2010.040206.
  37. Prince Stephen, ‘Through the Looking Glass: Philosophical Toys and Digital Visual Effects,’ Projections 4 (2) 2010: 19-40. doi: 10.3167/proj.2010.040203.
  38. Schaefer, Alexandre, Frédéric Nils, Xavier Sanchez, and Pierre Philippot, ‘Assessing
    the Effectiveness of a Large Database of Emotion-eliciting Films: A New Tool for
    Emotion Researchers,’ Cognition and Emotion 24 (7) 2010: 1153-1172. doi 10.1080/02699930903274322.
  39. Schramm, Holger, and Werner Wirth, ‘Exploring the Paradox of Sad-film Enjoyment: The Role of Multiple Appraisals and Meta-appraisals,’ Poetics 38 (3) 2010: 319-335. doi: 10.1016/j.poetic.2010.03.002.
  40. Schwan, Stephan, and Sermin Ildirar, ‘Watching Film for the First Time: How Adult Viewers Interpret Perceptual Discontinuities in Film,’ Psychological Science 21 (7) 2010: 970-976. doi: 10.1177/0956797610372632.
  41. Smith, Murray, ‘Feeling Prufish,’ Midwest Studies in Philosophy 34 (1) 2010: 261-279. doi: 10.1111/j.1475-4975.2010.00202.x.
  42. Smuts, Aaron, ‘The Ghost Is the Thing: Can Reactions to Fiction Reveal Belief?,’ Midwest Studies in Philosophy 34 (1) 2010: 219-239. doi: 10.1111/j.1475-4975.2010.00201.x.
  43. Stevens, Catherine, Heather Winskel, Clare Howell, Lyne-Marine Vidal, Cyril Latimer, Josephine Milne-Home, ‘Perceiving Dance: Schematic Expectations Guide Experts’ Scanning of a Contemporary Dance Film,’ Journal of Dance Medicine and Science 14 (1) 2010: 19-25.
  44. Suckfüll, Monica, ‘Films That Move Us: Moments of Narrative Impact in an Animated Short Film,’ Projections 4 (2) 2010: 41-63. doi: 10.3167/proj.2010.040204.
  45. Visch, Valentijn, Ed S. Tan, and Dylan Molenaar, ‘The Emotional and Cognitive Effect of Immersion in Film Viewing,’ Cognition and Emotion 24 (8) 2010: 1439-1445. doi: 10.1080/02699930903498186.
  46. Wessel, Ineke, Rafaële J.C. Huntjens, and Johan R.L. Verwoerd, ‘Cognitive Control and Suppression of Memories of an Emotional Film,’ Journal of Behaviour Therapy and Experimental Psychiatry 41 (2) 2010: 83-89. doi: 10.1016/j.jbtep.2009.10.005.
  47. Yanal, Robert J., ‘Hybrid Truths and Emotion in Film,’ Midwest Studies in Philosophy 34 (1) 2010: 180-189. doi: 10.1111/j.1475-4975.2010.00212.x.
  48. Zacks, Jeffrey M., Nicole K. Speer, Khena M. Swallow, and Corey J. Maley, ‘The Brain’s Cutting-room Floor: Segmentation of Narrative Cinema,’ Frontiers in Human Neuroscience 4 2010: 168. doi: 10.3389/fnhum.2010.00168.
  49. Zhu, Xun, Xiaoying Wang, Carolyn Parkinson, Chengxu Cai, Song Gao, and Peicheng Hu, ‘Brain Activation Evoked by Erotic Films Varies with Different Menstrual Phases: An fMRI Study,’ Behavioural Brain Research 206 (2) 2010: 279-285. doi: 10.1016/j.bbr.2009.09.027.