What do the processes of sensation and perception entail? - Chapter 1

• Sensation and perception are central to, and often precede, almost all aspects of human behaviour and thought. There are many practical applications of our increased understanding of sensation and perception.
• Gustav Fechner invented several clever methods for measuring the relationship between physical changes in the world and consequent psychological changes in observers. These methods remain in use today. Using Fechner's methods, researchers can measure the smallest levels of a stimulus that can be detected (absolute threshold) and the smallest differences that can be detected (difference thresholds, or just noticeable differences).
• A more recent development for understanding performance - signal detection theory - permits us to simulate changes in the perceiver (e.g.. internal noise and biases) in order to understand perceptual performance better.  
• We learn a great deal about perception by understanding the biological structures and processes involved. One early observation - the doctrine of specific nerve energies -expresses the fact that people are aware only of the activity of their nervous systems. For this reason, what matters is which nerves are stimulated, not how they are stimulated. The central nervous system reflects specialisations for the senses, from cranial nerves to areas of the cerebral cortex involved in perception.
• The essential activities of all neurons, including those involved in sensory processes, are chemical and electrochemical. Neurons communicate with each other through neurotransmitters, molecules that cross the synapse from the axon of one neuron to the dendrite of the next. Nerve impulses are electrochemical; voltages change along the axon as electrically charged sodium and potassium ions pass in and out of the membranes of nerve cells.
• Recordings of individual neurons enable us to measure the lowest level of a stimulus required for a neuron to fire (absolute threshold). Both the rate and the timing pattern of neuronal firing provide additional information about how the brain encodes stimuli in the world.
• Neuroimaging methods have revolutionised the study of sensation and per- ception by allowing us to study the brain in healthy, living human observers. Useful methods include electroencephalography (EEG), magnetoencephalog- raphy (MEG), positron emission tomography (PET), and functional magnetic resonance imaging (fMRI). Each comes with its own combination of temporal and spatial properties, making one method suitable for researching some questions and other methods more suitable for other questions.
• Computational models have become more useful than ever for revealing ways through which sensation and perception develop through experience. What all of these models have in common is that the world has predictability that can be exploited to make the senses more attuned to the world.
• Deep learning algorithms are powerful tools that can take in vast amounts of data (e.g.. pictures) and categorise it (That's a cat). Such tools may become important in applications like autonomous driving or detecting cancer in medical images.

How can our eyes perceive light? - Chapter 2

• This chapter provided some insight into the complex journey light must take for us to see stars and other spots of light. The path of the light was traced from a distant star through the eyeball and to its absorption by photoreceptors and its transduction into neural signals. In subsequent chapters we'll learn how those signals are transmitted to the brain and translated into the experience of perception.
• Light, on its way to becoming a sensation (a visual sensation, that is), can be absorbed, scattered, reflected, transmitted, or refracted. It can become a sensation only when it's absorbed by a photoreceptor in the retina.
• Vision begins in the retina, when light is absorbed by rods or cones. The retina is like a minicomputer that transduces light energy into neural energy.
• The high degree of convergence in the retinal periphery ensures high sensitivity to light but poor acuity.
• The low degree of convergence in the fovea ensures high acuity but poor sensitivity to light.
• The one-to-one pathways between cones and ganglions exist only in the fovea and account for why images are seen most clearly when they fall on this part of the retina.
• The visual system deals with large variations in overall light intensity by (a) regulating the amount of light entering the eyeball, (b) using different types of photoreceptors in different situations, and (c) effectively throwing away photons we don't need. 
• The retina sends information to the brain via ganglion cells, neurons whose axons make up the optic nerves. Retinal ganglion cells have centre-surround receptive fields and are concerned with changes in contrast (the difference in Intensity between adjacent bits of the scene).
• Age-related macular degeneration (AMD) is a disease associated with aging that affects the macula. The leading cause of visual loss among the elderly in the United States, AMD gradually destroys sharp central vision, making it difficult to read, drive, and recognise faces.
• Retinitis pigmentosa (RP) is a family of hereditary diseases characterised by the progressive death of photoreceptors and degeneration of the pigment epithelium. In the most common form of the disease, patients first notice vision problems in their peripheral vision and under low-light conditions- situations in which rods play the dominant role in collecting light.
• Several exciting developments are aimed at restoring sight in individuals with blinding retinal diseases.

How do people perceive spatial figures? - Chapter 3

• In this chapter we followed the path of image processing from the eyeball to the brain. Neurons in the cerebral cortex translate the array of activity signalled by retinal ganglion cells into the beginnings of forms and patterns. The primary visual cortex is organised into thousands of tiny computers, each responsible for determining the orientation, width, colour, and other characteristics of the stripes in one small portion of the visual field. In Chapter 4 we will continue this story by seeing how other parts of the brain combine the outputs from these minicomputers to produce a coherent representation.
• Perhaps the most important feature of image processing is the remarkable transformation of information from the circular receptive fields of retinal ganglion cells to the elongated receptive fields of the cortex.
• Cortical neurons are highly selective along a number of dimensions, includ- ing stimulus orientation, size, direction of motion, and eye of origin.
• Neurons with similar preferences are often arranged in columns in the primary visual cortex.
• Selective adaptation provides a powerful, noninvasive tool for learning about stimulus specificity in human vision.
• The human visual cortex contains pattern analysers that are specific to spatial frequency and orientation.

How does object recognition work? - Chapter 4

• A series of extrastriate visual areas continue the work of visual processing. Emerging from V1 (primary visual cortex, striate cortex) are two broad streams of processing: one going into the temporal lobe and the other into the parietal lobe. The temporal pathway seems specifically concerned with what a stimulus might be. This chapter follows that pathway. The parietal where pathway will be considered in later chapters.
• After early visual processes extract basic features from the visual input, it is the job of mid-level vision to organise these features into the regions, surfaces, and objects that can, in turn, serve as input to object recognition and scene-understanding processes.
• Perceptual committees' serve as an important metaphor in this chapter. The idea is that many semi-independent processes are working on the input at the same time. Different processes may come to different conclusions about the presence of an edge or the relationship between two elements in the input. Under most circumstances, we see the single conclusion that the committees settle upon. Bayesian models are one way to formalise this process of finding the most likely explanation for input. Deep neural networks may be a way to build members/parts of the committee.
• Multiple processes seek to carve the input into regions and to define the edges of those regions, and many rules are involved in this parsing of the Image. For example, image elements are likely to group together if they are similar in colour or shape, if they are near each other, or if they are connected. Many of these grouping principles were first articulated by members of the Gestalt school.
• Other, related processes seek to determine whether a region is part of a fore- ground figure (like this black O) or part of the background (like the white area around the O). These rules of grouping and figure-ground assignment are driven by an implicit understanding of the physics of the world. Thus, events that are very unlikely to happen by chance (e.g., two contours parallel to each other) are taken to have meaning. (Those parallel contours are likely to be part of the same figure.)
• The processes that divide visual input into objects and background have to deal with many complexities. Among these are the fact that parts of objects may be hidden behind other objects (occlusion) and the fact that objects themselves have structure. Is your nose an object or a part of a larger whole? What about glasses or hair or a wig?
• In addition to perceiving the shapes of objects and their parts, we are also adept at categorising the material that an object seems to be made of-glass, stone, cloth, and so on. We use material perception to estimate physical properties. What would the object feel like? Can it be grasped like a bottle, or would it slip through our fingers like sand?
• It is common to talk about the role of 'templates' in object recognition. The idea is that an object in the world is recognised when its image fits a particular representation in the brain in the way that a key fits a lock. It has always been hard to see how naïve template models could work, because of the astronomical number of templates required: we might need one 'lock' for every object in every orientation in every position in the visual field.
• Deep neural networks (DNNs) are modern efforts to create computer algo: rithms that can categorise objects very well. Unlike a literal, lock-and-key template, a DNN tries to match an image with a complex pattern of activity in a network that arises when the network encounters an object in a specific category. This allows the network to categorise an infinite set of images as 'cat', 'coffee cup', and so on.
• Faces are a special case of object processing. Viewpoint is very important. Upright faces are much easier to recognise than inverted faces. Moreover, some regions of the brain seem to be specifically interested in faces. Different regions may be important for different aspects of face processing ("Who is it?' versus 'How is he feeling?").

How do we observe color? - Chapter 5

• Probably the most important fact to know about colour vision is that lights and surfaces look coloured because a particular distribution of wavelengths of light is being analysed by a particular visual system. Colour is a mental phenomenon, not a physical phenomenon. Many animal species have some form of colour vision. It seems to be important for identifying possible mates, possible rivals, and good things to eat. Colour vision has evolved sev eral times in several different ways in the animal kingdom.
• Rod photoreceptors are sensitive to low (scotopic) light levels. Humans have only one type of rod photoreceptor; it yields one 'number' for each location in the visual field. Our rods can support only a one-dimensional representation of colour, from dark to light. Thus, human scotopic vision is achromatic vision.
• Humans have three types of cone photoreceptors, each having a different sen- sitivity to the wavelengths of light. Cones operate at brighter light levels than rods, producing three numbers at each location; the pattern of activity over the different cone types defines the colour. Some animals have many more types of photoreceptors, but we know rather little about their colour experience.
• If two regions of an image produce the same response in the three cone types, they will look identical; that is, they will be metamers. And they will look identi cal even if the physical wavelengths coming from the two regions are different.
• In additive colour mixture, two or more lights are mixed. Adding a light that looks blue to a light that looks yellow will produce a light that looks white (if we pick the right blue and yellow). In subtractive colour mixture, the filters, paints, or other pigments that absorb some wavelengths and reflect others are mixed. Mixing a typical blue paint and a typical yellow paint will subtract most long and short wavelengths from the light reflected by the mixture, and the result will look green.
• Colour blindness is typically caused by the congenital absence or abnormality of one cone type - usually the L- or M-cone, usually in males. Most colour-blind individuals are not completely blind to differences in wavelength. Rather, their colour perception is based on the outputs of two cone types instead of the normal three.
• A single type of cone cannot be used, by itself, to discriminate between wavelengths of light. To enable discrimination, information from the three cones is combined to form three cone-opponent processes. In the first, cones sensitive to long wavelengths (L-cones) are pitted against medium-wavelength (M) cones to create an L-M process that is rough- ly sensitive to the redness or greenness of a region. In the second cone-opponent process, L- and M-cones are pitted against short-wavelength (S) cones to create an (L+ M) - S process roughly sensitive to the blueness or yellowness of a region. The third process is sensitive to the overall brightness of a region.
• Colour appearance is arranged around opponent colours: red versus green, and blue versus yellow. This colour opponency involves further reprocessing of the cone signals from cone-opponent processes into colour-opponent processes.
• The visual system tries to disentangle the properties of surfaces in the world from the properties of the illuminants, even though surface and illuminant information are combined in the input to the eyes. Mechanisms of colour constancy use implicit knowledge about the world to correct for the influence of different illuminants and to keep that strawberry looking red under a wide range of conditions.

How does the perception of space and binocular vision work? - Chapter 6

• Reconstructing a three-dimensional world from two non-Euclidean, curved, two-dimensional retinal images is one basic problem faced by the brain.
• A number of monocular cues provide information about three-dimensional space. These include occlusion, various size and position cues, aerial perspective, linear perspective, motion cues, accommodation, and convergence.
• Having two eyes is an advantage for a number of reasons, some of which have to do with depth perception. It is important to remember, however, that it is possible to reconstruct the three-dimensional world from a single two-dimensional image. Two eyes have other advantages over just one: expanding the visual field, permitting binocular summation, and providing redundancy if one eye is damaged.
• Having two laterally separated eyes connected to a single brain also provides us with important information about depth through the geometry of the small differences between the images in each eye. These differences, known as binocular disparities, give rise to stereoscopic depth perception.
• Random dot stereograms show that we don't need to know what we're seeing before we see it in stereoscopic depth. Binocular disparity alone can support shape perception.
• Stereopsis has been exploited to add, literally, depth to entertainment-from nineteenth-century photos to twenty-first-century movies. It has also served to enhance the perception of information in military and medical settings.
• The difficulty of matching an image element in one eye with the correct element in the other eye is known as the correspondence problem. The brain uses several strategies to solve the problem. For example, it reduces the initial complexity of the problem by matching large blobs' in the low-spatial-frequency information before trying to match every high-frequency detail.
• Single neurons in the primary visual cortex and beyond have receptive fields that cover a region in three-dimensional space, not just the two-dimensional image plane. Some neurons seem to be concerned with a crude in-front/behind judgement. Other neurons are concerned with more precise, metrical depth perception.
• When the stimuli on corresponding loci in the two eyes are different, we experience a continual perceptual competition between the two eyes, known as binocular rivalry. Rivalry is part of the effort to make the best guess about the current state of the world based on the current state of the input.
• All of the various monocular and binocular depth cues are combined (unconsciously) according to what prior knowledge tells us about the probability of the current event. Making the wrong guess about the cause of visual input can lead to illusions. Bayes' theorem is the basis of one type of formal understanding of the rules of combination.
• Stereopsis emerges suddenly at about 4 months of age in humans, and it can be disrupted through abnormal visual experience during a critical period early in life.

How does attention and the perception of a scene work? - Chapter 7