Category Archives: Cognitive Film Theory

More Visual Illusions

I like visual illusions – though I must admit that the rotating snakes (Figure 1) from Professor Akiyoshi Kitaoka’a illusion pages makes me feel somewhat queasy.

Figure 1 Rotating snakes from Akiyoshi’s illusion pages (click on the image for a larger version or got to to see the illusion in all its glory).

You can find other versions of this illusion and many others at Akiyoshi’s illusion pages here, along with research papers that discuss the psychological basis of the illusions he features. An fMRI study of the above illusion is Kuriki I, Ashida H, Murakami I, and Kitaoka A 2008 Functional brain imaging of the Rotating Snakes illusion by fMRI, Journal of Vision 8 (10): 16, 1-10, and can be accessed here.

The Daily Cognition has twenty visual illusions here. presents a study of illusions from 1922 – Matthew Luckiesh’s Visual Illusions: Their Causes, Characteristics and Applications – in its entirety.

An interesting introduction to the role of visual illusions in psychological research is David Eagleman’s article on how the study of visual illusions has guided neuroscience research: Eagleman DM 2001 Visual illusions and neurobiology, Nature Reviews Neuroscience 2: 920-926.

Richard Gregory, who died in May of last year, conducetd a large amount of research on visual perception and illusions, and the website dedicated to his memory – and featuring some of his papers on illusion and perception – can be accessed here.

Another researcher in visual illusions is Cornelia Fermüller, and her website can be found here, and includes examples of illusions and her research on a computational theory of optical illusions in video sequences and stereo images. One paper worth reading is Ogale AS, Fermüller C, and Aloimonos Y 2005  Motion segmentation using occlusions, IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (6): 988-992.

We examine the key role of occlusions in finding independently moving objects instantaneously in a video obtained by a moving camera with a restricted field of view. In this problem, the image motion is caused by the combined effect of camera motion (egomotion), structure (depth), and the independent motion of scene entities. For a camera with a restricted field of view undergoing a small motion between frames, there exists in general a set of 3D camera motions compatible with the observed flow field even if only a small amount of noise is present, leading to ambiguous 3D motion estimates. If separable sets of solutions exist, motion-based clustering can detect one category of moving objects. Even if a single inseparable set of solutions is found, we show that occlusion information can be used to find ordinal depth, which is critical in identifying a new class of moving objects. In order to find ordinal depth, occlusions must not only be known, but they must also be filled (grouped) with optical flow from neighboring regions. We present a novel algorithm for filling occlusions and deducing ordinal depth under general circumstances. Finally, we describe another category of moving objects which is detected using cardinal comparisons between structure from motion and structure estimates from another source (e.g., stereo).

This paper from Alex Holcombe looks at the illusion of motion perception in the cinema:

Holcombe AO 2009 Seeing slow and seeing fast: two limits on perception, Trends in Cognitive Neuroscience 13 (5): 216-221. [This paper contains links to movies as part of the paper’s supplementary materials].

Video cameras have a single temporal limit set by the frame rate. The human visual system has multiple temporal limits set by its various constituent mechanisms. These limits seem to form two groups. A fast group comprises specialized mechanisms for extracting perceptual qualities such as motion direction, depth and edges. The second group, with coarse temporal resolution, includes judgments of the pairing of color and motion, the joint identification of arbitrary spatially separated features, the recognition of words and high-level motion. These temporally coarse percepts might all be mediated by high-level processes. Working at very different timescales, the two groups of mechanisms collaborate to create our unified visual experience.

Mel Slater’s paper looks at why we experience a sense of immersion in artificially created environments.

Slater M 2009 Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments, Philosophical Transactions of the Royal Society B 364 (1535): 3549-3557.

In this paper, I address the question as to why participants tend to respond realistically to situations and events portrayed within an immersive virtual reality system. The idea is put forward, based on the experience of a large number of experimental studies, that there are two orthogonal components that contribute to this realistic response. The first is ‘being there’, often called ‘presence’, the qualia of having a sensation of being in a real place. We call this place illusion (PI). Second, plausibility illusion (Psi) refers to the illusion that the scenario being depicted is actually occurring. In the case of both PI and Psi the participant knows for sure that they are not ‘there’ and that the events are not occurring. PI is constrained by the sensorimotor contingencies afforded by the virtual reality system. Psi is determined by the extent to which the system can produce events that directly relate to the participant, the overall credibility of the scenario being depicted in comparison with expectations. We argue that when both PI and Psi occur, participants will respond realistically to the virtual reality.

A different approach to the nature of immersion in visual perception of animated images can be found in this paper from Kenny Chow and Fox Harrell:

Chow KKN and Harrell DF 2009 Material-based imagination: embodied cognition in animated images, Cognition and Creativity, Digital Arts and Culture 2009, Arts Computation Engineering, UC Irvine, California,;jsessionid=A526BE7A0733A59BB004CD0AC05A9EB6.

Drawing upon cognitive science theories of conceptual blending and material anchors, as well as recent neuroscience results regarding mirror neurons, we argue that animated visual graphics, as embodied images whose understanding relies on our perceptual and motor apparati, connect both material and mental notions of images. Animated visual images mobilize a reflective process in which material-based imaginative construction and elaboration can take place. We call this process as “material-based imagination,” in contrast to the general notion of imagination as purely a mental activity. This kind of imagination is pervasive in today’s digitally mediated environments. By analyzing a range of digital artifacts from computer interfaces to digital artworks, we show the important role of imaginative blends of concepts in making multiple levels of meaning, including visceral sensation and metaphorical narrative imagining, to exemplify expressiveness and functionality. The implications of these analyses collectively form a step toward an embodied cognition approach to animation phenomena and toward recentralizing understanding of artistic and humanistic production in cognitive research.

Finally, the 2011 finalists for the Illusion of the Year contest can be found here.

Empirical studies in film style IV

It has been a while since I listed some research on the empirical analysis of film style – I could have sworn I did a post on this just before christmas, but apparently not.

First, a couple of general papers that outline the principles of video content analysis (VCA) and the research that has been done in this area. This piece (here) is a set of power point slides by Alan Hanjalic (see below), in which he summarises the goals of VCA, its applications, and the different approaches that have been adopted by researchers. A literature survey of work in VCA is given in the following paper:

Brezeale D and Cook DJ 2008 Automatic video classification : a survey of the literature, IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 38 (3): 416-430. DOI: 10.1109/TSMCC.2008.919173.

Earlier posts on the empirical analysis of film style can be accessed here, here, and here.

The papers referred to below all cover the relationship between emotion, style, and video content.

Arifin S and Chueng PYK 2006 User attention based arousal content modelling, IEEE International Conference on Image Processing, 8 November 2006, Atlanta, Georgia, USA.


The affective content of a video is defined as the expected amount and type of emotion that are contained in a video. Utilizing this affective content will extend the current scope of application possibilities. The dimensional approach to representing emotion can play an important role in the development of an affective video content analyzer. The three basic affect dimensions are defined as valence, arousal and control. This paper presents a novel FPGA-based system for modeling the arousal content of a video based on user saliency and film grammar. The design is implemented on a Xilinx Virtex-II xc2v6000 on board a RC300 board.

The poster for this paper can be accessed here.

Hanjalic A 2006 Extracting moods from pictures and sounds, IEEE Signal Processing Magazine 23 (2): 90-100. DOI: 10.1109/MSP.2006.1621452.

From the introduction:

Intensive research efforts in the field of multimedia content analysis in the past 15 years have resulted in an abundance of theoretical and algorithmic solutions for extracting the content-related information from audiovisual signals. The solutions proposed so far cover an enormous application scope and aim at enabling us to easily access the events, people, objects, and scenes captured by the camera, to quickly retrieve our favorite themes from a large music video archive (e.g., a pop/rock concert database), or to efficiently generate comprehensive overviews, summaries, and abstracts of movies, sports TV broadcasts, surveillance, meeting recordings, and educational video material. However, what about the task of finding exciting parts of a sports TV broadcast or funny and romantic excerpts from a movie? What about locating unpleasant video clips we would be reluctant to let our children watch? This article considers how we feel about the content we see or hear. As opposed to the cognitive content information composed of the facts about the genre, temporal content structure (shots, scenes) and spatiotemporal content elements (objects, persons, events, topics) we are interested in obtaining the information about the feelings, emotions, and moods evoked by a speech, audio, or video clip. We refer to the latter as the affective content, and to the terms such as “happy ” or “exciting ” as the affective labels of an audiovisual signal.

Hanjalic A and Xu L 2005 Affective video content and representation modelling, IEEE Transactions on Multimedia 7 (1): 143-154. DOI: 10.1109/TMM.2004.840618.


This paper looks into a new direction in video content analysis – the representation and modelling of affective video content. The affective content of a given video clip can be defined as the intensity and type of feeling or emotion (both are referred to as affect) that are expected to arise in the user while watching that clip. The availability of methodologies for automatically extracting this type of video content will extend the current scope of possibilities for video indexing and retrieval. For instance, we will be able to search for the funniest or the most thrilling parts of a movie, or the most exciting events of a sport program. Furthermore, as the user may want to select a movie not only based on its genre, cast, director and story content, but also on its prevailing mood, the affective content analysis is also likely to contribute to enhancing the quality of personalizing the video delivery to the user. We propose in this paper a computational framework for affective video content representation and modelling. This framework is based on the dimensional approach to affect that is known from the field of psychophysiology. According to this approach, the affective video content can be represented as a set of points in the two-dimensional (2-D) emotion space that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect).We map the affective video content onto the 2-D emotion space by using the models that link the arousal and valence dimensions to low-level features extracted from video data. This results in the arousal and valence time curves that, either considered separately or combined into the so-called affect curve, are introduced as reliable representations of expected transitions from one feeling to another along a video, as perceived by a viewer.

Machajdik J and Hanbury A 2010 Affective image classification using features inspired by psychology and art theory, ACM Multimedia Conference 25-29 October 2010, Firenze, Italy.


Images can affect people on an emotional level. Since the emotions that arise in the viewer of an image are highly subjective, they are rarely indexed. However there are situations when it would be helpful if images could be retrieved based on their emotional content. We investigate and develop methods to extract and combine low-level features that represent the emotional content of an image, and use these for image emotion classification. Specifically, we exploit theoretical and empirical concepts from psychology and art theory to extract image features that are specific to the domain of artworks with emotional expression. For testing and training, we use three data sets: the International Affective Picture System (IAPS); a set of artistic photography from a photo sharing site (to investigate whether the conscious use of colors and textures displayed by the artists improves the classification); and a set of peer rated abstract paintings to investigate the influence of the features and ratings on pictures without contextual content. Improved classification results are obtained on the International Affective Picture System (IAPS), compared to state of the art work.

This paper does not relate specifically to film, but I include it anyway becuase it is interesting to read alongside the other papers listed here and in the context of cognitive film theory. The pdf linked to for this paper is over 10MB, so it may be quite slow to download.

Soleymani M, Chanel G, Kierkels JJK, and Pun T 2008 Affective characterization of movie scenes based on multimedia content analysis and user’s physiological emotional responses, IEEE International Symposium on Multimedia, 15-17 December 2008, Berkeley, California, USA [Abstract only].


In this paper, we propose an approach for affective representation of movie scenes based on the emotions that are actually felt by spectators. Such a representation can be used for characterizing the emotional content of video clips for e.g. affective video indexing and retrieval, neuromarketing studies, etc. A dataset of 64 different scenes from eight movies was shown to eight participants. While watching these clips, their physiological responses were recorded. The participants were also asked to self-assess their felt emotional arousal and valence for each scene. In addition, content-based audio- and video-based features were extracted from the movie scenes in order to characterize each one. Degrees of arousal and valence were estimated by a linear combination of features from physiological signals, as well as by a linear combination of content-based features. We showed that a significant correlation exists between arousal/valence provided by the spectator’s self-assessments, and affective grades obtained automatically from either physiological responses or from audio-video features. This demonstrates the ability of using multimedia features and physiological responses to predict the expected affect of the user in response to the emotional video content.

Yoo HW and Cho SB 2007 Video scene retrieval with interactive genetic algorithm, Multimedia Tools and Applications 34 (3): 317-336. DOI: 10.1007/s11042-007-0109-8.


This paper proposes a video scene retrieval algorithm based on emotion. First, abrupt/gradual shot boundaries are detected in the video clip of representing a specific story. Then, five video features such as “average colour histogram,” “average brightness,” “average edge histogram,” “average shot duration,” and “gradual change rate” are extracted from each of the videos, and mapping through an interactive genetic algorithm is conducted between these features and the emotional space that a user has in mind. After the proposed algorithm selects the videos that contain the corresponding emotion from the initial population of videos, the feature vectors from them are regarded as chromosomes, and a genetic crossover is applied to those feature vectors. Next, new chromosomes after crossover and feature vectors in the database videos are compared based on a similarity function to obtain the most similar videos as solutions of the next generation. By iterating this process, a new population of videos that a user has in mind are retrieved. In order to show the validity of the proposed method, six example categories of “action,” “excitement,” “suspense,” “quietness,” “relaxation,” and “happiness” are used as emotions for experiments. This method of retrieval shows 70% of effectiveness on the average over 300 commercial videos.

Finally, a report from a couple of years ago that appeared in IEEE Spectrum about a jacket that lets you “feel the movies” to add a sense of touch to the emotional events in a film.

The jacket contains 64 independently controlled actuators distributed across the arms and torso. The actuators are arrayed in 16 groups of four and linked along a serial bus; each group shares a microprocessor. The actuators draw so little current that the jacket could operate for an hour on its two AA batteries even if the system was continuously driving 20 of the motors simultaneously.

So what can the jacket make you feel? Can it cause a viewer to feel a blow to the ribs as he watches Bruce Lee take on a dozen thugs? No, says Lemmens. Although the garment can simulate outside forces, translating kicks and punches is not what the actuators are meant to do. The aim, he says, is investigating emotional immersion.

The article can be accessed here.

Cognitive film theory: 2010 bibliography update

Last January, I posted my bibliography of books, articles, etc., on the subject of cogntive film theory and which can be accessed here. This post includes articles published in this area during 2010. The list is not exhaustive, but it is accurate.

  1. Addis, Michael, and Morris B. Holbrook, ‘Consumers’ Identification and Beyond: Attraction, Reverence, and Escapism in the Evaluation of Films,’ Psychology and Marketing 27 (9) 2010: 821–845. doi: 10.1002/mar.20359.
  2. Bartsch, Anne, ‘Vivid Abstractions: On the Role of Emotion Metaphors in Film Viewers’ Search for Deeper Insight and Meaning,’ Midwest Studies in Philosophy 34 (1) 2010: 240-260. doi: 10.1111/j.1475-4975.2010.00198.x.
  3. Bartsch, Anne, Markus Appel, and Dennis Storch, ‘Predicting Emotions and Meta-Emotions at the Movies: The Role of the Need for Affect in Audiences’ Experience of Horror and Drama,’ Communication Research 37 (2) 2010: 167-190. doi: 10.1177/0093650209356441.
  4. Blumstein, Daniel T., Richard Davitan, and Peter D. Kaye, ‘Do Film Soundtracks Contain Nonlinear Analogues to Influence Emotion?,’ Biology Letters 6 (6) 2010: 751-754. doi: 10.1098/rsbl.2010.0333.
  5. Bordwell, David, ‘The Part-time Cognitivist: A View from Film Studies,’ Projections 4 (2) 2010: 1-18. doi: 10.3167/proj.2010.040202.
  6. Branigan, Edward, ‘Soundtracks in Mind,’ Projections 4 (1) 2010: 41-67. doi: 10.3167/proj.2010.040104.
  7. Bruun Vaage, Margrethe, ‘Fiction Film and the Varieties of Empathic Engagement,’ Midwest Studies in Philosophy 34 (1) 2010: 158-79. doi: 10.1111/j.1475-4975.2010.00200.x.
  8. Carroll, Noël, ‘Movies, the Moral Emotions, and Sympathy,’ Midwest Studies in Philosophy 34 (1) 2010: 1-19. doi: 10.1111/j.1475-4975.2010.00197.x.
  9. Currie, Gregory, ‘Bergman and the Film Image,’ Midwest Studies in Philosophy 34 (1) 2010: 323-339. doi: 10.1111/j.1475-4975.2010.00199.x.
  10. Cutting, James E., Jordan E. DeLong, and Christine E. Noether, ‘Attention and the Evolution of Hollywood Film,’ Psychological Science 21 (3) 2010: 432-439. doi: 10.1177/0956797610361679.
  11. Dadlez, E.M., ‘Seeing and Imagination: Emotional Response to Fictional Film,’ Midwest Studies in Philosophy 34 (1) 2010: 120-135. doi: 10.1111/j.1475-4975.2010.00194.x.
  12. Dennis, Tracy A., and Beylul Solomon, ‘Frontal EEG and Emotion Regulation: Electrocortical Activity in Response to Emotional Film Clips is Associated with Reduced Mood Induction and Attention Interference Effects,’ Biological Psychology 85 (3) 2010: 456-464. doi: 10.1016/j.biopsycho.2010.09.008.
  13. Deutsch, Stephan, ‘Psycho and the Orchestration of Anxiety,’ The Soundtrack 3 (1) 2010: 53-66. doi: 10.1386/st.3.1.53_1.
  14. Eder, Jens, ‘Understanding Characters,’ Projections 4 (1) 2010: 16-40. doi: 10.3167/proj.2010.040103
  15. Elliot, Paul, ‘The Eye, The Brain, The Screen: What Neuroscience Can Teach Film Theory,’ Excursions 1 (1) 2010: 1-16.
  16. Feagin, Susan L., ‘Film Appreciation and Moral Insensitivity,’ Midwest Studies in Philosophy 34 (1) 2010: 20-33. doi: 10.1111/j.1475-4975.2010.00197.x.
  17. Friend, Stacie, ‘Getting Carried Away: Evaluating the Emotional Influence of Fiction Film,’ Midwest Studies in Philosophy 34 (1) 2010: 77-105. doi:  10.1111/j.1475-4975.2010.00196.x.
  18. Gaut, Berys, ‘Empathy and Identification in Film,’ Midwest Studies in Philosophy 34 (1) 2010: 136–157. doi: 10.1111/j.1475-4975.2010.00211.x.
  19. Giovannelli, Alessandro, ‘Cognitive Value and Imaginative Identification: The Case of Kubrick’s Eyes Wide Shut,’ The Journal of Aesthetics and Art Criticism 68 (4) 2010: 355–366. doi: 10.1111/j.1540-6245.2010.01430.x.
  20. Grau, Christophe, ‘American History X, Cinematic Manipulation, and Moral Conversion,’ Midwest Studies in Philosophy 34 (1) 2010: 52-76. doi: 10.1111/j.1475-4975.2010.00210.x.
  21. Grodal, Torben, ‘High on Crime Fiction and Detection,’ Projections 4 (2) 2010: 64-85. doi: 10.3167/proj.2010.040205.
  22. Harold, James, ‘Mixed Feelings: Conflicts in Emotional Responses to Film,’ Midwest Studies in Philosophy 34 (1) 2010: 280-294. doi:  10.1111/j.1475-4975.2010.00209.x.
  23. Hirose, Yoriko, Alan Kennedy, and Benjamin W. Tatler, ‘Perception and Memory Across Viewpoint Changes in Moving Images,’ Journal of Vision 10 (4) 2010: 2. doi:10.1167/10.4.2.
  24. Hopkins, Robert, ‘Moving because Pictures? Illusion and the Emotional Power of Film,’ Midwest Studies in Philosophy 34 (1) 2010: 200-218. doi: 10.1111/j.1475-4975.2010.00208.x.
  25. Kauppi, Jukka-Pekka, Iiro P. Jääskeläinen, Mikko Sams, and Jussi Tohka, ‘Inter-subject Correlation of Brain Hemodynamic Responses During Watching a Movie: Localization in Space and Frequency,’ Frontiers in Neuroinformatics 4 2010: 5. doi: 10.3389/fninf.2010.00005.
  26. Kim, Seahwa, ‘The Rationality of Emotion toward Fiction,’ Midwest Studies in Philosophy 34 (1) 2010: 106-119. doi: 10.1111/j.1475-4975.2010.00203.x.
  27. Kim, Sue J. ‘Anger, Cognition, Ideology: What Crash Can Show Us about Emotion,’ Image and Narrative 11 (2) 2010:
  28. Laine, Tarva, ‘The Diving Bell and the Butterfly as an Emotional Event,’ Midwest Studies in Philosophy 34 (1) 2010: 295-305. doi: 10.1111/j.1475-4975.2010.00207.x.
  29. Lamarque, Peter, and Peter Goldie, ‘Whimsicality in the Films of Eric Rohmer,’ Midwest Studies in Philosophy 34 (1) 2010: 306-322. doi: 10.1111/j.1475-4975.2010.00206.x.
  30. Livingstone, Paisley, ‘On the Appreciation of Cinematic Adaptations,’ Projections 4 (2) 2010: 104-127. doi: 10.3167/proj.2010.040207.
  31. Matravers, Derek, ‘Why We Should Give Up on the Imagination,’ Midwest Studies in Philosophy 34 (1) 2010: 190-199. doi: 10.1111/j.1475-4975.2010.00205.x.
  32. Michel, Eva, ‘The Role of Individual Differences in Cognitive Skills in Children’s Learning Through Film,’ Journal of Media Psychology 22 (3) 2010: 105-113. doi: 10.1027/1864-1105/a000015.
  33. Nixon, Lizze, ‘I Focalize, You Focalize, We All Focalize Together: Audience Participation in Persepolis,’ Image and Narrative 11 (2) 2010:
  34. Oliver, Mary Beth, and Tilo Hartmann, ‘Exploring the Role of Meaningful Experiences in Users’ Appreciation of “Good Movies,”’ Projections 4 (2) 2010: 128-150. doi: 10.3167/proj.2010.040208.
  35. Planitnga, Carl, ‘“I Followed the Rules, and They All Loved You More”: Moral Judgment and Attitudes toward Fictional Characters in Film,’ Midwest Studies in Philosophy 34 (1) 2010: 34-51. doi: 10.1111/j.1475-4975.2010.00204.x.
  36. Planitnga, Carl, ‘Affective Incongruity and The Thin Red Line,’ Projections 4 (2) 2010: 86-103. doi: 10.3167/proj.2010.040206.
  37. Prince Stephen, ‘Through the Looking Glass: Philosophical Toys and Digital Visual Effects,’ Projections 4 (2) 2010: 19-40. doi: 10.3167/proj.2010.040203.
  38. Schaefer, Alexandre, Frédéric Nils, Xavier Sanchez, and Pierre Philippot, ‘Assessing
    the Effectiveness of a Large Database of Emotion-eliciting Films: A New Tool for
    Emotion Researchers,’ Cognition and Emotion 24 (7) 2010: 1153-1172. doi 10.1080/02699930903274322.
  39. Schramm, Holger, and Werner Wirth, ‘Exploring the Paradox of Sad-film Enjoyment: The Role of Multiple Appraisals and Meta-appraisals,’ Poetics 38 (3) 2010: 319-335. doi: 10.1016/j.poetic.2010.03.002.
  40. Schwan, Stephan, and Sermin Ildirar, ‘Watching Film for the First Time: How Adult Viewers Interpret Perceptual Discontinuities in Film,’ Psychological Science 21 (7) 2010: 970-976. doi: 10.1177/0956797610372632.
  41. Smith, Murray, ‘Feeling Prufish,’ Midwest Studies in Philosophy 34 (1) 2010: 261-279. doi: 10.1111/j.1475-4975.2010.00202.x.
  42. Smuts, Aaron, ‘The Ghost Is the Thing: Can Reactions to Fiction Reveal Belief?,’ Midwest Studies in Philosophy 34 (1) 2010: 219-239. doi: 10.1111/j.1475-4975.2010.00201.x.
  43. Stevens, Catherine, Heather Winskel, Clare Howell, Lyne-Marine Vidal, Cyril Latimer, Josephine Milne-Home, ‘Perceiving Dance: Schematic Expectations Guide Experts’ Scanning of a Contemporary Dance Film,’ Journal of Dance Medicine and Science 14 (1) 2010: 19-25.
  44. Suckfüll, Monica, ‘Films That Move Us: Moments of Narrative Impact in an Animated Short Film,’ Projections 4 (2) 2010: 41-63. doi: 10.3167/proj.2010.040204.
  45. Visch, Valentijn, Ed S. Tan, and Dylan Molenaar, ‘The Emotional and Cognitive Effect of Immersion in Film Viewing,’ Cognition and Emotion 24 (8) 2010: 1439-1445. doi: 10.1080/02699930903498186.
  46. Wessel, Ineke, Rafaële J.C. Huntjens, and Johan R.L. Verwoerd, ‘Cognitive Control and Suppression of Memories of an Emotional Film,’ Journal of Behaviour Therapy and Experimental Psychiatry 41 (2) 2010: 83-89. doi: 10.1016/j.jbtep.2009.10.005.
  47. Yanal, Robert J., ‘Hybrid Truths and Emotion in Film,’ Midwest Studies in Philosophy 34 (1) 2010: 180-189. doi: 10.1111/j.1475-4975.2010.00212.x.
  48. Zacks, Jeffrey M., Nicole K. Speer, Khena M. Swallow, and Corey J. Maley, ‘The Brain’s Cutting-room Floor: Segmentation of Narrative Cinema,’ Frontiers in Human Neuroscience 4 2010: 168. doi: 10.3389/fnhum.2010.00168.
  49. Zhu, Xun, Xiaoying Wang, Carolyn Parkinson, Chengxu Cai, Song Gao, and Peicheng Hu, ‘Brain Activation Evoked by Erotic Films Varies with Different Menstrual Phases: An fMRI Study,’ Behavioural Brain Research 206 (2) 2010: 279-285. doi: 10.1016/j.bbr.2009.09.027.

Memory and experience

This week some papers on memory and experience in the cinema from a cognitive perspective.

One of the interesting things to note is that if you search Google scholar using the terms “viewer memory cinema” you get a lot of references to books and articles on the status of film as memory and cinema as cultural memory; whereas if you search for “viewer memory advertising” the results returned focus on how people experience and remember adverts (and the obvious economic consequences of this). Insights from research into advertising can be useful to film studies, and if we take a step beyond film studies we can find a good deal of empirical research that looks at how people organise and remember their experience of films. This is especially the case when we look at the research on cinema advertising, where we find many studies of how audiences respond to and recall what they have seen on the screen, but which is largely absent from work on cognitive film theory.

As ever, the version linked to may not be the final published version.

An interesting paper recently published by Frontiers in Human Neuroscience looks at how viewer’s organise their experience of a film by segmenting it into meaningful events by measuring brain activity. This research harks back to work in the 1970s by Carroll and Bever on segmentation in narrative cinema (Carroll JM and Bever TG 1976 Segmentation in narrative cinema, Science 191 (4231): 1053-1055).

Zacks JM, Speer NK, Swallow KM, and Maley CJ (2010) The brain’s cutting-room floor: segmentation of narrative cinema, Frontiers in Human Neuroscience 4: 168. doi: 10.3389/fnhum.2010.00168.

Observers segment ongoing activity into meaningful events. Segmentation is a core component of perception that helps determine memory and guide planning. The current study tested the hypotheses that event segmentation is an automatic component of the perception of extended naturalistic activity, and that the identification of event boundaries in such activities results in part from processing changes in the perceived situation. Observers may identify boundaries between events as a result of processing changes in the observed situation. To test this hypothesis and study this potential mechanism, we measured brain activity while participants viewed an extended narrative film. Large transient responses were observed when the activity was segmented, and these responses were mediated by changes in the observed activity, including characters and their interactions, interactions with objects, spatial location, goals, and causes. These results support accounts that propose event segmentation is automatic and depends on processing meaningful changes in the perceived situation; they are the first to show such effects for extended naturalistic human activity.

Although not directly related to the cinema (though films are mentioned), this paper also provides evidence for the way in which veiwer’s segment scenes:

Kurby CA and Zacks JM 2008 Segmentation in the perception and memory of events, Trends in Cognitive Science 12: 729-79.

People make sense of continuous streams of observed behavior in part by segmenting them into events. Event segmentation seems to be an ongoing component of everyday perception. Events are segmented simultaneously at multiple timescales, and are grouped hierarchically. Activity in brain regions including the posterior temporal and parietal cortex and lateral frontal cortex increases transiently at event boundaries. The parsing of ongoing activity into events is related to the updating of working memory, to the contents of long-term memory, and to the learning of new procedures. Event segmentation might arise as a side effect of an adaptive mechanism that integrates information over the recent past to improve predictions about the near future.

Both these papers come from Jeff Zack’s Dynamic Cognition Laboratory at Washington University, St. Louis, MO., and his webpage can be accessed here.

This next piece seems almost quaint now, with its tales of students who did not own a CD in the late-1980s, but provides some evidence for the way in which a viewer understands adverts in different ways.

Mick DG 1992 Levels of subjective comprehension in advertising processing and their relations to ad perceptions, attitudes and memories, Journal of Consumer Research 18 (4): 411-424.

Two fundamental orientations toward message comprehension have appeared un advertising research: the traditional objective view, which applies to the accuracy criterion to conceptualize and evaluate comprehension, and the subjective view, which applies other criteria related to the individual comprehender and the actual experience of the message. This article develops a framework for four levels of subjective comprehension on the basis of an elaboration criterion. Comprehension levels are hypothesized to differ in the relations to ad perceptions, attitudes, and memory. Results from an empirical study provide initial support for the framework, including new theoretical insights and explanatory ability beyond the objective orientation. Discussion focuses on implication for advertising theory and consumer research.

Lang A, Zhou S,Schwartz N, Bolls, PD, and Potter, RF 2000 The effects of edits on arousal, attention, and memory for television messages: when an edit is an edit can an edit be too much?, Journal of Broadcasting and Electronic Media 44 (1): 94-109.

This study examines the effect of the rate of edits (camera changes in the same visual scene) on viewers’ arousal and memory. The rate of edits varied from slow to very fast. Results show that as the rate of edits increases physiological arousal, self-reported arousal, and memory increase. It is suggested that edits can increase attention to and encoding of television message content without significantly increasing the cognitive load of the message.

An interesting paper looking at cinema advertsing is this piece from Hong Kong Baptist University, which focussed on how different groups experienced and recalled cinema advertising and what factors (screen size, stereo sound, etc.) affected those experiences. It reveals some interesting results: in Hong Kong, women are more likely to enter a theatre before the advertising than men, and so advertising directed at this group is likely to be more effective. I know of no similar study in the UK or the US, but I’m sure some data will exist somewhere – and if it doesn’t, then it should and there is a PhD here for someone.

Prendergast G and Chan LW 2003 Cinema Advertising in Hong Kong, BRC Working Papers, School of Business, Hong Kong Baptist University.

Cinema advertising offers a relatively less cluttered environment for advertisers to present their message to a captive audience. However, little is known about its effectiveness, especially in countries such as Hong Kong (a country that is relatively underdeveloped in terms of cinema adspend statistics). Building on the work of Ewing, Du Plessis and Foster (2001) and Dunnett and Hoek (1996), insights into perceptions of cinema advertising in Hong Kong were obtained from a survey of 150 interviewees. Different from previous studies which utilized dela yed recall, this study interviewed audience members immediately after they had viewed a particular movie. Results showed that cinema advertising exposure and recall rates were significantly related to various demographic variables, especially gender and age. Furthermore, the level of recall was found to be correlated with various situational stimuli in the cinema, such as the larger than life screen, Dolby stereo sound, the silent environment, comfortable seats, and audience members’ expectations to focus on the screen. Based on these findings, recommendations for cinema managers and advertisers are made.

One the major criticisms abelled at the recent Bond movies is that they are so stuffed with product placement that they often appear to be little more than glorified adverts. Recently, television programmes in the UK were given the go ahead to include more product placement as a way of increasing advertising revenues. Clearly, then it would be useful to ave some research int he effectvieness of product placement in films -and we have this research:

Bressoud E, Lehu, J-M, and Russel CA 2008 Integrating placement and audience characteristics to assess the recall of product placements in film: findings from a field study, 7th International Conference on Research in Advertising (ICORIA), 27-28 June 2008, Antwerp, Belgium.

This research incorporates into a single model characteristics of product placements in films and characteristics of the consumers and their viewing environment to assess the memorability of the placements. Eleven movies containing a total of 98 placements of varied characteristics were coded. 3,532 individuals who viewed a DVD rental of one of these movies at home completed a questionnaire on the following day. The questionnaire included audience viewing characteristics as well as a free recall measure of placements. The results reveal important insights into the variables thataffect, positively or negatively, the day after recall of products placed in movies.

Empirical research on narratives

A few weeks ago I published a post that looked at mathematical models of narrative comprehension that could be used for empirical research into how viewers understand narrative films. I also complained that the reason there was a lack of empirical research on narrative comprehension was simply because no-one in film studies has undertaken such research, whereas in many other disciplines an empirically based approach is fundamental. You can read the earlier post here. This week I include some abstracts and some links to papers that do look at the relationship between agents and narratives empirically. An interesting aspect that should also be noted in many of these papers is the way in which concepts of film style have moved into other media, while thinking about narrative in virtual environments and interactive fiction provides new ways of thinking about narrative in the cinema.

As ever, the versions of these papers linked may not be the final published version.

Bizocchi J 2005 Run, Lola, Run: film as a narrative database, Media in Transition 4: The Work of Stories, May 6-8, 2005, MIT, Cambridge, MA. [I’ll have more to say about this paper and the topic of database narratives at a later date in a piece on paraconsistency in narrative cinema].

Clarke A and Mitchell G 2001 Film and the development of interactive narrative, International Conference on Virtual Storytelling: Using Virtual Reality Technologies for Storytelling, 27-28 September 2001, Avignon, France.

This paper explores narration in film and in videogames/virtual environments/interactive narratives. Particular attention is given to their use of the continuity of time, space and action and this is used as a means of classifying different types of work. The authors argue that the creators of these videogames etc. need to have more authorial presence and that this can only be done through abandoning their traditional reliance on the continuity of time, space and action.

Johnson K and Bizzocchi J forthcoming Lost Cause: an interactive film project, Journal of the International Digital Media and Arts.

The paper describes the design, the aesthetics, and the experience of the interactive film Lost Cause. The film is examined from several theoretical perspectives: cinematic roots, narrative construction, interface design, and new media artifact. Lost Cause extends the complex plot structure used by filmmakers such as Altman or Tarentino into an explicitly interactive format. The plot has three interrelated and synchronous threads which are represented in a multiscreen user interface. It culminates in an ending determined by the history of user navigation choices. The paper analyzes the work to reveal critical insights into database narrative, expressive interface design, user agency, and the construction of micronarrative.

Marsh T, Nitsche M, Liu W, Chung P, Bolter JD, and Cheok AD 2008 Film informing design for contemplative gameplay, Sandbox Symposium, 9-10 August 2008 Los Angeles, California.

Borrowing from film and filmmaking styles, techniques and devices that manipulate spectators’ attention and experience, this paper proposes an approach to inform design of games and gameplay to manipulate player’s focus of attention and encourage contemplation — in design features, characters, story elements, etc. or even break the player’s engaged attention in the game/virtual world altogether — to provide meaning, experience and opportunities for learning. Focusing on film styles alternative to the continuity style of Hollywood filmmaking, we discuss examples of design for contemplative gameplay in game-based learning environments/serious games, machinima and augmented and mixed reality games in previous, current and future projects. We propose that one goal of game design is to establish a rhythm between contemplation and engagement, and the appropriate rhythm is determined largely by a game’s genre, platform and/or narrative.

May J and Barnard PJ 1995 Cinematography and interface design, in K Nordby, PH Helmersen, DJ Gilmore, and SA Arnesen (eds.) Human-Computer Interaction: Interact’95. London: Chapman and Hall: 26-31. [NB: there isn’t a direct URL for this paper, but if you google the title you should find the pdf version easy enough].

Interface designers are increasingly relying on craft based approaches to compensate for a perceived lack of relevant theory. One such source is cinematography, where film-makers succeed in helping viewers follow the narrative across cuts which change the information on the screen. Cinematography has evolved over the last century, and its rules of thumb cannot be applied directly to interface design. We analyse film-makers’ techniques with a cognitive theory (ICS) and show that they work by preserving thematic continuity across cuts. Expressing this theoretically allows us to extrapolate away from film, applying it to screen changes in interface design.

Nath S 2004 Narrativity in user action: emotion and temporal configurations of narrative, 4th International Conference on Computational Semiotics for Games and New Media, 14-16 September 2004, Split, Croatia.

One of the core problems in Narrative Intelligence is maintaining the narrative nature of event sequences that emerge owing to user participation. This paper challenges the common premises and assumptions about the nature of human action and experience that underlie common approaches to finding a solution to the problem of narrative structuration. An in-depth analysis of the temporality of human action and experience provides important indicators on how the problem can be approached. It is argued that user emotion is not just a by-product of narrative structure, but a critical factor in maintaining narrativity. Finally, it is indicated as to how patterning of emotions can regulate user action and the creation of a subjective experience.

Rowe JP and Lester JC 2010 Modelling user knowledge with dynamic Bayesian networks in interactive narrative environments

Recent years have seen a growing interest in interactive narrative systems that dynamically adapt story experiences in response to users’ actions, preferences, and goals. However, relatively little empirical work has investigated runtime models of user knowledge for informing interactive narrative adaptations. User knowledge about plot scenarios, story environments, and interaction strategies is critical in a range of interactive narrative contexts, such as mystery and detective genre stories, as well as narrative scenarios for education and training. This paper proposes a dynamic Bayesian network approach for modelling user knowledge in interactive narrative environments. A preliminary version of the model has been implemented for the CRYSTAL ISLAND interactive narrative-centred learning environment. Results from an initial empirical evaluation suggest several future directions for the design and evaluation of user knowledge models for guiding interactive narrative generation and adaptation.

This paper by Rowe and Lester is from the Intellimedia group at North Carolina State University, which publishes a wide range of papers on human-computer interaction, virtual learning environments, and narrative interaction. Their website can be accessed here.


Modelling narrative comprehension in film studies

In an essay titled ‘Can scientific models of theorizing help film studies’, Malcolm Turvey (2005: 25) asks why ‘is there a lack of systematic empirical research in film theory if the explanatory principles governing the cinema are like the explanatory principles governing natural phenomena?’

Turvey’s answer is that theories in the humanities do not need empirical validation in the same way as a theory in the sciences because ‘cinema is a human creation’ and is ’embedded in human practices and institutions,’ and researchers do not need to empirically validate their theory because they already know a great deal about such practices and institutions. Humanistic theories clarify our knowledge about things we already know, whereas scientific theories are generate new knowledge about subjects we have no prior knowledge about.

One of the examples of just such a humanistic theory accepted without validation cited by Turvey is David Bordwell’s (1985) constructivist account of the viewer’s comprehension of narrative. However, empirical research on narrative comprehension is to be found sociology, psychology, linguistics, and many other disciplines; but this is not the case in film studies. It is not because there is something special about film theories that means they do not require empirical validation – it is simply because no one has ever tried. Turvey does not address the fundamental reason for the lack empirical research in film studies: namely, that the vast majority of film scholars simply do not know how to design and conduct a piece of empirical research. There is a lack of basic empirical research skills in film studies, but this is unsurprising given that theory courses dominate film degrees while empirically based courses are few and far between. It is of course important that students study theory, if only so that they may have an understanding of the history of film studies as an academic discipline; but this should be accompanied with an emphasis on empirical research. Until this is generally the case, empirical validation of film theories will not be achieved.

We may have not trouble understanding a narrative film, but there are many questions about narrative comprehension in the cinema that can only be addressed through empirical research, including:

  • Which pieces of evidence in a narrative do viewer’s consider salient?
  • What hypotheses do viewers form to explain narrative events, and what role do they play in creating expectations?
  • How do different viewers weight the same piece of evidence in their reasoning, and why do they differ?
  • To what extent is the weighting of evidence determined by regimes of generic and cultural verisimilitude?
  • Is the viewer sceptical, withholding belief until the end of a narrative when its conclusion becomes apparent; or is she credulous, committing belief early on in a narrative only for her assumptions to be overturned?
  • What impact does the way in which evidence is presented to viewer have on her belief? For example, does it make a difference if a piece of evidence is presented visually with accompanying dialogue from a character, if it is presented visually only, or if it is presented as dialogue only?
  • Does the viewer strive to achieve the local or global coherence of a narrative?
  • Does the viewer make inferences that are unnecessary to the successful comprehension of the narrative?

Although we have no problem in understanding narratives, we do not have answers to these questions. If, as Turvey claims, film theories concern ‘what human beings already know and do‘ (25), it begs the question how do we know what human beings already know and do? How could we determine what human beings already know and do if we do not research (empirically) what it is that human beings know and do? We do not have answers to the above questions because no one has done the research, not because humanistic theories can be accepted without empirical validation. As the reader will soon learn, the plausibility of a theory is the not the same as truth.

Bordwell D 1985 Narration in the Fiction Film. London: Methuen.

Turvey M 2005 Can scientific models of theorizing help film theory?, in TE Wartenberg and A Curran (eds.) The Philosophy of Film: Introductory Texts and Readings. Malden, MA: Blackwell Publishing: 21-32.

In order to conduct research into narrative comprehension in the cinema we need some methods that will allow us to model the reasoning of viewers, and some methods are discussed in the my paper ‘Modelling inference in the comprehension of cinematic narratives.’ The abstract is below, and the pdf file can be downloaded beneath that. There are also some links to papers that provide the background theory to the concepts used in this article below.

Modelling inference in the comprehension of cinematic narratives


The purpose of this paper is to outline some models of inference that can be used in describing and analysing the behaviour of real viewers in comprehending a cinematic narrative. The viewer’s processes of inference making in the cinema involve the framing of hypotheses about the world of the narrative which may be overturned by subsequent information and are, therefore, nonmonotonic. The viewer’s reasoning can be modelled mathematically, and two approaches are discussed here: the use of Bayes’ Theorem to represent and update the subjective and conditional probabilities of an agent is summarised, and Peter Abell’s Bayesian approach to the sociological understanding of narratives is outlined; and the transferable belief model developed by Phillipe Smets, in which the beliefs of an agent are represented by belief functions that do not assume an underlying probability distribution. In understanding narratives and the understanding of narratives it is useful to represent information visually, and an analytic and synthetic method of representing inference via Wigmore charts is outlined.

Nick Redfern – Modelling inference in the comprehension of cinematic narratives

In reading this paper a little set theory would not go amiss, and the basics can be found here.

Some of the references  in this paper can be accessed freely online, and I have included links to many of them below. They are organised under broad categories of relevance to the paper. (NB: some of these links may not be to the final published versions of these papers).

Bayesian statistics

Goldstein M 2006 Subjective Bayesian analysis: principles and practice, Bayesian Analysis 1 (3): 403-420.

Evidence: method, philosophy, and theory

The UCL web page devoted to the research of evidence and inquiry has many interesting pieces that describes methods and philosophies of evidence that are worth reading with a view to exploring cognition and reasoning in the cinema, and may be accessed here.

Man as intuitive statistician

Brunswick E 1943 Organismic achievement and environmental probability, Psychological Review 50 (3): 255-272.

Kelley HH 1973 The process of causal attribution, American Psychologist 28: 107-128.

Peterson CR and Beach LR 1967 Man as intuitive statistician, Psychological Bulletin 68 (1): 29-46.

Cosmides L and Tooby J 1996 Are humans good intuitive statisticians after all? Rethinking some conclusions from the literature on judgment under uncertainty, Cognition 58: 1-73. (NB: an online version of this paper is available but there is no URL associated with it, but you can find it by searching for the title easily enough).

Narrative action theory

Abell P 2007 Narratives, Bayesian narratives, and narrative actions, Sociologica 3:

Nonmonotonic reasoning

Brewka G, Niemalä I, and Truszczyński M 2007 Nonmonotonic reasoning, in F van Harmelen, V Lifschitz, and B Porter (eds.) Handbook of Knowledge Representation. Amsterdam: Elsevier: 239-284.

Josephson JR 1994 Conceptual analysis of abduction, in JR Josephson and SG Josephson (eds.) Abductive Inference: Computation, Philosophy, Technology. New York: Cambridge University Press: 5-30.

Reiter R 1980 A logic for default reasoning, Artificial Intelligence 13: 81-132.

Transferable belief model

Smets P 1990a The combination of evidence in the transferable belief model, IEEE-Pattern Analysis and Machine Intelligence 12 (5): 447-458.

Smets P 1991 About updating, in BD D’Ambrosio, P Smets, and PP Bonisone (eds.) Proceedings of the 7th Conference on Uncertainty in Artificial Intelligence. San Mateo, CA: Morgan Kaufman: 378-385.

Smets P 2002 Decision making in a context where uncertainty is represented by belief functions, in RP Srivastava and TJ Mock (eds.) Belief Functions in Busines Decisions. New York: Physica-Verlag: 17-61.

Smets P and Kennes R 1994 The transferable belief model, Artificial Intelligence 66 (2): 191–234.

Srivastava RP 1997 Decision making under ambiguity: a belief-function perspective, Archives of Control Sciences 6 (1-2): 5-27.

Visual illusions for film studies

This post contains a number of link to various web sites devoted to visual illusions and some papers on visual illusions that are of interest film researchers. (NB: the papers linked to may not be the final published versions).

Possibly the best site devoted to illusions is Michael Bach’s 87 Optical Illusions and Visual Phenomena, which has an impressive array of very nicely presented illusions. As a researcher on visual perception, Bach has published many papers on how we experience the world including Bach M, Poloschek CM (2006) Optical Illusions, Advances in Clinical Neuroscience and Rehabilitation 6 (2): 20–21, which  provides a short general overview.

George Mather is a researcher at the University of Sussex, who has developed the two-stroke apparent motion illusion, and published on this topic: Mather, G (2006) Two-stroke: a new illusion of visual motion based on the time course of neural responses in the human visual system, Vision Research 46: 2015-2018; and Mather G, Challinor KL (2009) Psychophysical properties of two-stroke apparent motion, Journal of Vision 9 (1): 28.

Other papers on similar illusions are Conway BR, Kitaoka A, Yazdanbakhsh A, Pack CC, Livingstone MS (2005) Neural basis for a powerful static motion illusion, The Journal of Neuroscience 25 (23): 5651-5656; and

Mather’s two-stroke illusion won second place at the ‘Illusion of the Year’ contest in 2005. For information on the other visual illusions, the contest’s web site is here.

Another good site is at the University Of Massachusetts Psychology Department and is maintained by David T. Landrigan.

The wagon-wheel illusion in particular is of interest to film researchers as it raises the question of why we experience motion. (The wagon wheel illusion demonstration at Bach’s website is particularly good). Short-range apparent motion is the traditional explanation for the experience of motion in the cinema:

The illusion of continuous motion is called apparent motion to distinguish it from ‘real’ motion, which is perceived when an object moves continuously across a viewer’s visual field. When Sir Laurence Olivier appears to be fencing in a film, he is in apparent motion, whereas a person walking across the theatre in front the screen is in real motion (Ramachandran & Anstis 1986: 102).

Ramachandran and Anstis’s argument is based on an ontological distinction between ‘real’ motion and ‘apparent’ motion, and that there is a one-to-one correspondence between stimulus and experience. Thus a continuous stimulus results in the experience of ‘real’ motion, while a discrete stimulus results in ‘apparent motion’ (Ramachandran ,V.S., and Anstis, S.M. (1986) The perception of apparent motion, Scientific American 254 (6): 102-109.) But what if our perception is discrete, so that we experience the world as a sequence of snapshots? The one-to-one correspondence between stimulus and experience implied by the ‘real’/’apparent’ distinction is not relevant if all our percepts are discrete, and the ontology of the stimulus (continuous/discrete) is irrelevant to our experience. In simple terms, all our perceptions would be discrete irrespective of the nature of the stimulus, and there would be no difference between watching Olivier on film or someone walking in front of the screen. The viewer would experience motion in the cinema because he experiences motion. This problem is raised in the following papers, most of which cite the cinema as a direct example of discrete perception:

Andrews T, Purves D (2005) The wagon-wheel illusion in continuous light, Trends in Cognitive Sciences 9 (6): 261-263.

Purves D, Paydarfar JA, Andrews TJ (1996) The wagon wheel illusion in movies and reality, Proceedings of the National Academy of Sciences 93 (8): 3693-3697.

van Rullen R, Koch C (2003) Is perception discrete or continuous?, Trends in Cognitive Sciences 7 (5): 207-213.

van Rullen R, Reddy L, Koch C (2005) Attention-driven discrete sampling of motion perception, Proceedings of the National Academy of Sciences 102 (14): 5291-5296.

van Rullen R, Reddy L, Koch C (2006) The continuous wagon wheel illusion is associated with changes in electroencephalogram power at ~13 Hz, The Journal of Neuroscience 26 (2): 502-507.

Van Rullen R, Pascual-Leone A, Batelli L (2008) The continuous wagon wheel illusion and the ‘when’ pathway of the right parietal lobe: a repetitive transcranial magnetic stimulation study, Public Library of Science One 3 (8): e2911.

A challenge to the argument for discrete perception argument in the wagon wheel illusion can be found in Klein K, Holcombe AO, Eagleman DM (2004) Illusory motion reversal is caused by rivalry, not by perceptual snapshots of the visual field, Vision Research 44: 2653-2658; and Klein K, Eagleman DM (2008) Evidence against the temporal subsampling account of illusory motion reversal, Journal of Vision 8 (4): 13.

Yves Gallifret deals with the history of retinal persistence and cinema in an English language essay from Comptes Rendues Biologies: Gallifret Y (2006) Visual persistence and cinema?, Compte Rendues Biologies 329 (5-6): 369-385.

An article on a similar subject is Paul St. George’s piece on chronophotography: St. George, P (2009) Using chronophotography to replace Persistence of Vision as a theory for explaining how animation and cinema produce the illusion of continuous motion, Animation Studies 4: 17-26.

Cognitive film theory: a bibliography

This weeks post is a bibliography of materials on the subject of cognitive film theory I have amassed on and off over the past few years. Although it contains some 355 items it is neither exhaustive nor up to date, although it should be accurate (barring any changes in the URLs for web-based resources). I’m sure most of what is there is well-known to those interested in this area, but there is almost certainly something you will not have come across before.

The file can be downloaded here as a pdf: Nick Redfern – CognitiveFilmTheoryBibliography1-19.

Finally, to bring to your attention an interesting article I came across recently on the subject of Luchino Visconti and Federico Fellini in Frontiers of Neurology and Neuroscience and their series on the impact of neurological disorders on famous artists, which looks at the impact of strokes on the creativity of two of Italy’s greatest filmmakers.

Dieguez, S., Assal, G., Bogousslavsky, J. (2007) Visconti and Fellini: from left social neorealism to right-hemisphere stroke, Frontiers of Neurology and Neuroscience 22: 44-74.

The acclaimed Italian directors Luchino Visconti and Federico Fellini had very different life trajectories that led them to become major figures in the history of cinema. Similarities, however, can be found in their debuts with the neorealist genre, their personalities, creative styles and politicocultural involvement, and ultimately in the neurological disease that struck them at the end of their careers. Both suffered a right-hemispheric stroke that left them hemiplegic on the left side. We review their life and career to put that event into perspective, and then discuss its aftermath for both artists in the light of our current knowledge of right-hemispheric functions. Visconti showed a tremendous resilience following the accident and managed to direct several films and plays as an infirm, whereas Fellini had to put an end to his career but still was able to display his talents to the neuropsychologists that treated him. A speculative account is given of the links between right-hemispheric symptomatology and the premorbid personality of these highly prolific patients.

PMID: 17495505

The transferable belief model in film and games studies

This post is another draft paper, this time focusing on the difference between ergodic and non-ergodic texts and so is of relevance to games studies as well as film studies. I use an approach that I don’t think has been applied to either of fields yet: the transferable belief model, which is a mathematical theory of evidence. Hopefully soon I will be able to outline how this model will be of use in film studies in more depth, along with considering its relationship to Bayesian approaches to modelling viewer behaviour. The idea is to apply these models to the empirical analysis of the beliefs of real spectators so that it may be possible to make some statements about how we understand films that are more than theoretical but which have a solid evidential basis. The article can be downloaded as a pdf file here:Nick Redfern – Credal and pignistic reasoning in ergodic and non-ergodic texts.


This paper discusses the difference between ergodic and non-ergodic texts by considering the different levels of reasoning required of an agent in each case. The difference indentified between such texts is based on the distinction between credal and pignistic reasoning in the transferable belief model. It is argued that non-ergodic texts require an active agent to reason about the state of the world, and thus operate at the credal level; while ergodic texts require that the belief function of an agent be transformed into a probability function for the purposes of decision making, and therefore entail both credal and pignistic reasoning. The difference between ergodic and non-ergodic texts considered in these terms is illustrated through comparing narratives from the CSI: Crime Scene Investigation franchise.

Information transduction in the cinema

A version of this post was presented to the CCM Research group at the University of Central Lancashire on 21 February 2007. It is presented here with the first part – a (not entirely satisfactory) discussion of ecological approaches to film theory – missing, and a new introduction. The model in figure 1 now seems incomplete and needs further development, but I think it still has some uses as a basic description of information and perception in the cinema.

Wade and Swanston point out that in order to come to a full understanding of vision it is necessary to ‘include an appreciation of the neurophysical processes that are initiated by the activity of light on the receptors of the eye. These involve the modification of light energy into nerve impulses and their transmission to areas at the back of the brain where they are analysed’ (1991: 59). In my opinion, it is precisely this ‘appreciation of neurophysical processes’ that should form the basis of film theory – not least because we need to be able to account for own experiences of the cinema in formulating hypotheses about it. A cognitive approach to film theory allows theorists to build self-reflexivity into their research. The Society for Cognitive Studies of the Moving Image promotes research in this area, with a particular emphasis on viewer’s emotional experiences. David Bordwell has recently written on the latest research to emerge here and  here.

However, I have been repeatedly frustrated by the lack of a model of communication in the cinema that can account for the viewer’s perception of a film in terms of neurophysical processes. For example, Carl Plantinga (1999) has noted that ‘[o]ne of the least explored aspects of film and television is their sensory means of communication’ (239) – but then on the next page he asserts that ‘[c]learly, film directors use the human face to communicate information about the emotions of characters’ (240, original emphasis). These are the only two occasions Plantinga mentions communication – but he goes from a topic that is one of the ‘least explored’ to the self-evident clarity of the assertion that films communicate emotions to the viewer. Plantinga never defines what he means by communication or information, though we may infer that he means the transmission from screen to spectator of meaningful content about the emotional states of a particular character. However, as Thayer has pointed out, such communication is impossible:

The ways in which we traditionally conceive of communication – those being inadequate and untenable – stand as obstacles to more adequate and more potent ways of conceiving of communication … Those preconceptions, our traditional concepts of communication, are often insidious. ‘Communication is the “transfer of meaning”’ has an appealing ring to it. But since none of our receptors is capable of receiving ‘meaning,’ the notion of transfer is a flagrantly untenable one (Thayer 1979: 10).

The work of Paul Bach-y-Rita (2002, 2003) at the University of Wisconsin deserves special mention here. Bach-y-Rita and his fellow researchers have used televisual sensory substitution systems to restore the modality of sight to vision impaired individuals, and he has addressed the impact of restored sight and emotion content:

we found that while experienced blind TVSS subjects could perceive faces and printed images, they were very disappointed when perception was not accompanied by qualia: A Playboy centerfold carried no emotional message, and the face of a girl-friend or a wife created an unpleasant response since it did not convey an affective message. We consider this to be comparable to the lack of emotional contact of curse-words in a language that has been learned as an adult. It is possible that the emotional content could be developed over a long period of usage. On the other hand, a blind infant using a vision substitution system smiles when he recognizes a toy and reaches for it, and a blind 10-year-old child perceiving a flickering candle flame by means of a TVSS is enchanted (Bach-y-Rita et al. 2003: 293).

This quote is very suggestive for cognitive film theorists working in the area of emotion. It raises a fundamental question: what is the nature of communication in the cinema? This paper explores this question through looking at information and the different forms it takes in the viewer’s experience of a motion picture.

Information in the cinema

Francis Crick points out that there ‘is one fact about the brain that is so obvious it is seldom mentioned: it is attached to the rest of the body and communicates with it. The nervous system receives information only from the various transducers in the body’ (1994: 81). This principle has long history and may be traced back to Johannes Müller’s law of specific nerve energies, which states that it no matter how a sensory system is stimulated, the resulting sensation will always be of the type appropriate to that system (Müller 1826). For example, the stimulation of the optic nerve will result in visual sensation regardless of whether that stimulation is by flashing light, by electric shock, or by pressure on the eye (Norrsell et al. 1999); and this sensation is dependent upon the part of the brain in which the sensory pathways terminate and not the stimulus. Thus we ‘see with the brain, not the eyes’ (Bach-y-Rita et al. 2003: 285) as the images that pass through the pupil and are focussed on the retina go no further: ‘The sole source of output from the retina to the rest of the brain is the action potentials arising from the million or so ganglion cells’ (Bear et al. 2007: 300). The brain has no independent reference as to the cause of electro-chemical signals that are transmitted along the optic nerve because the ‘response of a nerve cell does not encode the physical nature of the agents that caused its response. Encoded is only “how much” at this point on my body, but not “what”’ (Foerster [1973] 2003: 214).

Images in the cinema are comprised of variable physical properties in the pattern of silver salts of the film’s emulsion (Enticknap 2005: 203), so that light projected through a film and reflected by a screen is energy in the form of electromagnetic radiation and is experienced by the viewer as changes in the intensity of light and colour, and its position in the frame (Read 1998: 1). This light carries no qualitative information about the environment – there is only quantitative data about the energetic properties of the light. As Foerster points out ([1972] 1981: 263): ‘the environment contains no information, the environment is as it is.’ The viewer, as a perceiving system, is capable of receiving light as energy as evolution has led to the development of a visual system that responds to changes in the quantitative properties of a stream of photons (e.g. hue, luminosity) without knowledge of the cause of such properties. The viewer, then, is open to energy (‘how much’) but is closed to information (‘what’) (Ashby 1956).

As sensory systems function by ‘transducing some type of environmental energy into a form that can be analysed by the cells in the central nervous system’ (Wade and Swanston 1991: 59), perception cannot be considered direct – it is mediated by the sensory and neurophysiological processes of the perceiver. Those processes begin with light being focussed onto the retina:

Light emitted by or reflected off objects in space can be imaged by the eye onto the retina. Light energy is first converted into membrane potential charges in the mosaic of photoreceptors. … photoreceptor membrane potential is converted into a chemical signal (the neurotransmitter glutamate), which is again converted into a membrane potential changes in the post-synaptic bi-polar horizontal cells. This process of electrical-chemical-electrical signalling repeats again and again, until the presence of light or dark or colour is finally converted to a charge in the action potential firing frequency of the ganglion cells (Bear et al. 2007: 306).

The light energy reflected by a cinema screen enters the viewer’s eye where it is converted into a pattern of stimulation, and is projected to the magnocellular and parvocellular layers of the lateral geniculate nucleus before being relayed to the visual cortex (Farah 2000). As quantitative information, the pattern of stimulation at the retina is syntactic (Shannon and Weaver 1949), and is transduced into the functional information by the firing of neurons in the visual cortex. There is a non-random correlation between these two types of information (Gulick [1980] 1990).

A perceiving system is not aware of this complex process of information transduction, only of the results of this process (Jackendoff 1987). Somehow – and this remains a mystery to philosophers, psychologists, and neuroscientists – a perceiver is conscious of these results and they are meaningful. The semantic information that is the content of consciousness is an emergent product of the mind, and cannot be reduced to syntactic relations: information only becomes meaningful when a perceiver is able to link it to information structures he or she already possesses (Stonier 1997). Such information, then, is highly context-dependent, and from this it must be concluded that the environment does not contain sufficient information to guide a perceiver’s behaviour. The question of meaning comes down to how a perceiving system generates units of experience and relates them to conceptual structures that from the basis for subsequent modes of acting and thinking (Redfern 2004). As semantic information cannot be distinguished from those modes it becomes pragmatic information when it is embedded in a social practice (Zoglauer 1996).

The transduction of light energy to syntactic to functional to semantic/pragmatic information creates a layered hierarchy of information levels in which higher information concepts depend on lower level information concepts but cannot be reduced to them (Zoglauer 1996). This relationship is represented in Figure 1. Though we are conscious only of information at the semantic/pragmatic level of this hierarchy it is essential that we include the lower information levels and do not marginalise the physical inputs to the viewer as a perceiving system or ignore the viewer’s neurophysiological processes of information transduction.


Figure 1 Information transduction in motion picture perception

Representation in the cinema

The concept of representation is central to both cognitive psychology and to the study of all media forms as well as more specifically the cinema. Representation is a term used in a wide variety of senses and can refer to any symbolic description of the world. Thus, from a psychological perspective, representation refers to mental images that correspond to objects that lie beyond a perceiver’s sensory systems; while in film studies, representation is used in reference to the reflection or distortion of the ‘real’ (Dyer 1985). Both these uses of representation are manifest in the application of cognitive models to the cinema: in watching, say, The Scarlet Pimpernel (Harold Young, 1934) the viewer has a mental image of the depiction of the Pimpernel as an English gentleman adventurer (Richards 1997) (see Figure 2).

If the above description of information transduction in motion picture perception is accepted then the concept of representation is of no use in accounting for the viewer’s experience of a film. As the viewer is, in informational terms, organisationally closed his or her conscious experience cannot be said to correspond to anything that exists independently of the viewer. As an organisationally closed system that interacts necessarily with its own states, the viewer has no external point of reference by which to judge the correspondence of mental images to the world: the viewer has no means of establishing a correspondence between his or her perception of The Scarlet Pimpernel and the film itself. If it is accepted that the viewer is organisationally closed then the sensory structure, patterns, or images he or she experiences are the viewer’s own construction, and the notion that they represent an aspect of the world has no empirical foundation. Glasersfeld (1995, 1999) argues that in the place of representation, we should use the term presentation, as this is closer to Kant’s deployment of vorstellung in ‘The Conflict of the Faculties,’ to refer to concepts that are generated by a perceiver: ‘the mind can only create only presentations of its own objects and not of the real things, that is, through these presentations and concepts, things cannot possibly be known as they might be in themselves’ (quoted in Glasersfeld 1995: 39-40). The qualitative aspects of the viewer’s experience are solely determined by the viewer, and his or her experience of The Scarlet Pimpernel is a mental presentation of the viewer that emerges as a result of a complex process of information transduction.


Figure 2 Leslie Howard as The Scarlett Pimpernel (1934)

The concept of constructions that do not correspond to an external reality does not imply epistemological solipsism. Lorenz (1941) argued that evolution has provided us with a perceptual system that allows us to operate in the absence of information about the ‘real world.’ This principle has been developed by evolutionary epistemologists (Campbell 1974) and radical constructivists (Glasersfeld 1995), who argue that constructions are adaptations that provide us with viable ways of thinking and acting in an environment (Sjölander 1999). This principle of adaptation is derived from the work of Jean Piaget (1937), who approached the construction of knowledge as a biologist. For Piaget, adaptation involves two complimentary and simultaneous processes: a cognising organism primarily seeks to organise experience in terms of the psychological structures (schemes) it already possesses, i.e., it seeks to assimilate experience; if the result of this process creates a perturbation the organism attempts to accommodate the error either by modifying an existing scheme or creating a new one. It is this balance between assimilation and accommodation that Piaget describes as adaptation. Knowledge is actively constructed, and is adapted to fit the environmental constraints that act on an organism in order to avoid internal contradictions and achieve equilibrium. Glasersfeld describes the principle of adaptation in radical constructivist thought:

[A]daptation is not an activity but the result of the elimination of all that is not adapted. Consequently, on the biological level, anything that manages to survive is ‘adapted’ to the environment in which it happens to find itself living.… Taken out of the biological context and applied to cognition, this means that ‘to know’ is not to possess true representations of reality, but rather to possess ways and means of acting and thinking that will allow one to attain the goals one happens to have chosen (2001: 39).

The key to evaluating competing knowledge claims, therefore, is not to seek to compare them to the ‘real world’ that cannot be known, but to assess their cognitive viability or functional fitness.

In observing a film, the viewer abstracts regularities from his or her conscious experience and seeks to fit those regularities into pre-existing information structures. In watching The Scarlet Pimpernel, the viewer identifies regularities in their experience of watching a film and links this semantic information to what he or she already knows about ‘Englishness’ and ‘gentlemanly behaviour.’ A viewer who has no understanding of these concepts will be unable to establish such connections, and as a consequence will interpret the film very differently. Perception is, to a significant extent, dependent upon the viewer’s prior experiences and knowledge, and is comprised of a sense-making activity that involves the building up of conceptual structures by linking new information to old. The result of this process cannot be regarded as being representational as the viewer has no means of accessing the film directly.


In adopting an evolutionary-constructivist approach it is possible to develop a model of information transduction in motion picture perception that is non-representational, ecologically viable, and takes as its starting point the viewer as a biological perceiving system. This approach leads to the conclusion that perception is not direct, and the highly detailed, coherent world that such a viewer experiences is an autopoietic construct of the viewer (Maturana and Varela 1980). The ‘impression of reality’ in the cinema may be accounted for as the viewer’s construction of his or her own experiential reality. The viewer does not pickup information from a film because, to paraphrase Foerster, the film contains no information; the film is as it is.


Ashby, W.R. (1956) An Introduction to Cybernetics. London: Chapman and Hall.

Bach-y-Rita, P. (2002) Sensory substitution and qualia, in A. Noe and E. Thompson (eds.) Vision and Mind: Selected Readings in the Philosophy of Perception. Cambridge, MA: MIT Press: 497-514.

Bach-y-Rita, P., M.E. Tyler, and K.A. Kaczmarek (2003) Seeing with the brain, International Journal of Human-Computer Interaction 15 (2): 285-295.

Bear, M.F., B.W. Connors, and M.A. Paradiso (2007) Neuroscience: Exploring the Brain, third edition. Philadelphia: Lippincott, Williams, and Williams.

Campbell, D.T. (1974) Evolutionary epistemology, in P.A Schilpp (ed.) The Philosophy of Karl R. Popper. LaSalle, IL: Open Court: 412-463.

Crick, F. (1994) The Astonishing Hypothesis. London: Simon and Schuster.

Dyer, R. (1985) Taking popular television seriously, in D. Lusted and P. Drummond (eds.) TV and Schooling. London: BFI: 41-46.

Enticknap, L. (2005) Moving Image Technology: From Zoetrope to Digital. London: Wallflower Press.

Farah, M.J. (2000) The Cognitive Neuroscience of Vision. Malden, MA: Blackwell.

Foerster, H. von ([1972] 1981) Notes on an epistemology of living things, Biological Computer Laboratory Report 9.3, University of Illinois; reprinted in Observing Systems. Seaside, CA: Intersystems Publishing: 258-271.

Foerster, H. von ([1973] 2003) On constructing a reality, in F.E. Preiser (ed.) Environmental Research Design, Volume 2. Stroudsburg, Dowden, Hutchinson & Ross: 35-46; reprinted in Understanding Understanding: Essays on Cybernetics and Cognition, New York, Springer: 211-228.

Glasersfeld, E. von (1995) Radical Constructivism: A Way of Knowing and Learning. London: RoutledgeFarmer.

Glasersfeld, E. von (1999) Piaget’s legacy: cognition as adaptive activity, in A. Riegler, M. Peschl, and A. von Stein (eds.) Understanding Representation in the Cognitive Sciences: Does Representation Need Reality? New York/Dordrecht: Kluwer Academic/Plenum Publishers: 283-287.

Glasersfeld, E. von (2001) The radical constructivist view of science, Foundations of Science 6 (1-3): 31-43.

Gordon, I.E. (1997) Theories of Visual Perception, second edition. Chichester: John Wiley & Sons.

Gulick, R. van ([1980] 1990) Functionalism, information, and content, Nature and System 2: 139-162; reprinted in W.G. Lycan (ed.) Mind and Cognition: A Reader. Oxford: Blackwell, 107-129.

Jackendoff, R. (1987) Consciousness and the Computational Mind. Cambridge, MA: MIT Press.

Lorenz, K. (1941) Kants Lehre von Apriorischen im Lichte gegenwärtiger Biologie. Die angeborene Formen möglicher Erfahrung, Blätter für deutsche Philosophie 15: 94-125.

Maturana, H.R., and F. Varela (1980) Autopoiesis and Cognition: The Realisation of the Living. Dordrecht: D. Reidel Publishing Company.

Müller, J. (1826) Zur vergleichenden Physiologie des Gesichtssinnes des Menschen und der Tiere. Leipzig: C. Knobloch.

Norrsell, U., S. Finger, and C. Lajonchere (1999) Cutaneous sensory spots and the ‘law of specific nerve energies:’ history and development of ideas, Brain Research Bulletin 48 (5): 457-465.

Piaget, J. (1937) La construction du réel chez l’enfant. Neuchatel: Delachaux et Niestle.

Plantinga, C. (1999) The scene of empathy and the human face in film, in C. Platinga and G.M. Smith (eds.) Passionate Views: Film, Cognition, and Emotion. Baltimore and London: The Johns Hopkins University Press: 239-255.

Read, R. (1998) The Essence of Communication Theory. London: Prentice Hall.

Redfern, N. (2004) Communication and meaning in the cinema, Constructivism in the Human Sciences 9 (2): 39-48.

Richards, J. (1997) Films and British National Identity: From Dickens to Dad’s Army. Manchester: Manchester University Press.

Shannon, C.E., and W. Weaver (1949) The Mathematical Theory of Communication. Urbana, Il.: University of Illinois Press.

Sjölander, S. (1999) On the evolution of reality – some biological prerequisites and evolutionary stages, Journal of Theoretical Biology 187 (4): 595-600.

Stonier, T. (1997) Information and Meaning: An Evolutionary Perspective. New York: Springer.

Thayer, L. (1979) Communication: sine qua non of the behavioural sciences, in R.W. Budd and B.D. Ruben (eds.) Interdisciplinary Approaches to Human Communication. Rochelle Park, NJ: Hayden: 7-31.

Wade, N.J., and M. Swanston (1991) Visual Perception: An Introduction. London and New York: Routledge.

Zoglauer, T. (1996) Can information be naturalised?, in K. Kornwachs and K. Jacoby (eds.) Information: New Questions to a Multidisciplinary Concept. Berlin: Akademie-Verlag: 187-207.