Timing is of critical importance for visual experiments both in terms of stimuli presentation and responses recording. Because Mac OS X is a multi-tasking operating system there is the inherent risk of inaccurate timing. Timing accuracy depends on many factors, among them: processor and memory usage, operating system version, type of video card, graphics driver, type of input/output device, type of display, as well as the way Psykinematix manages timing. Although not all experimental paradigms require accurate display and response timing, it is particularly important that the experimenter is aware of timing problems when they occur.
Despite the various sources of timing variability, Psykinematix does its best to provide proper timing in visual presentation and input/output communication, and more important Psykinematix reports any timing problem it can detect. By default, Psykinematix is configured for optimal timing in most conditions:
-
Psykinematix runs at the highest priority during OpenGL operations to minimize interruptions from other processes,
-
Psykinematix ensures that each frame rendering is synced to the vertical refresh of the display to prevent tearing effects (through the use of double buffering and buffers swap during the vertical retrace),
-
Psykinematix computes and stores before each stimulation interval all stimuli textures in video memory to minimize memory transfer between the CPU and the graphics card.
This configuration leads to sub-millisecond precision in terms of video timing when Psykinematix is not overloaded. Unreliable timing results otherwise in "missed frames" which Psykinematix detects and reports after each session along with some statistics about the frame duration.
Display, Graphics Card & OpenGL
Visual stimuli can be presented to the subjects using a cathode ray tube (CRT), a liquid crystal display (LCD) or a data projector. The video signal is encoded in a digital or analog format by the graphics card for a given frame rate and resolution so the display or projector interprets it correctly as a 2D video stream. In addition, the role of the graphics card is to create and store the digital representation of the display content elaborated by the running applications (operating system included). The graphics card generally contains the video memory (VRAM) dedicated to the storage of the currently presented video frame, possibly several frame buffers (in preparation of the next few frames to be presented), and visual patterns (called textures) that may be re-used multiple times in the same or different video frames. All graphics cards in Mac OS X computers use OpenGL as rendering engine to generate the display content and can be set up to automatically synchronize the frame presentation with the display refresh rate.
Differences between CRT and LCD displays
CRT Displays
Frame rate and resolution are adjustable on CRT displays (often referred as MultiSync). CRT displays have 0 ms of input lag (delay between the graphics card output and the time the stimulus is shown). CRT displays have been well characterized (Bach 1997, Pelli 1997, Brainard et al 2002) and became the unavoidable display technology for vision research for the last 30 years.
Unfortunately, CRT monitors suitable for vision research are not anymore manufactured: Sony Trinitron, NEC-Mitsubishi Diamondtron, LaCie and Iiyama CRT brands have been now all discontinued (only Viewsonic has still produced 160 Hz CRTs until recently). The only alternative technology widely available today is LCD. Despite the apparent attractive features of LCD displays, it is important to be aware of their shortcomings in the context of visual research, particularly regarding their timing properties.
LCD Displays
The frame rate for LCD displays is generally fixed (ie: not user-adjustable) or limited to a few ones (eg: 60, 75, 85 Hz). Their resolution is also fixed due to the limited number of physical pixels that composed their screen, which is referred as native resolution. The LCD resolution can nevertheless be scaled down to lower resolutions, but this is not recommended for psychophysical experiments as the non-native resolutions may introduce some visual artifacts such as blurriness or "blockiness" in the visual stimuli.
It is also important to note that most LCD displays present a significant input lag of several tens of milliseconds (corresponding to the duration of one or several video frames). Such large input lag is because most LCD monitors perform some video processing on the images before sending them to the display panel to compensate for blur and ghosting effects introduced by the pixel persistence for the entire duration of a frame (contrarily to CRT that uses short-persistence phosphors).
Moreover the timing specifications provided by LCD manufacturers are often very misleading because they provide the most impressive numbers without explaining how they are measured: for example, in 2009 LCD monitors are usually advertised as having response times of 2 or 5 millisecond. In the past, response time, typically in the range of several tens of milliseconds, was usually defined as the time needed to switch from black to white and back to black again (off-on-off or rise-and-fall). Now the response time is typically reported as the much shorter time to switch from one gray level to another gray level (gray-to-gray), and more often by selecting the two gray levels where the monitor performs best. Only a full specification of the gray-to-gray timing (ie: response time for each pair of gray levels) can provide a clear view of the LCD capability (see the Monitors section of the X-bit Labs web site for such technical review of commercially available LCD monitors, or the LCD Response Time and Motion Blur DisplayMate article).
Note that all internal displays (LCD) in Macintosh laptops have a fixed frame rate of 60 Hz and most stand-alone LCD monitors have a maximum frame rate between 75 Hz and 85 Hz. However there are a few new LCD monitors running at a true 120 Hz for stereoscopic applications with a low input lag, for example the Samsung SyncMaster 2233RZ or ViewSonic VX2268wm (note that a Dual Link DVI video output is required to drive these displays and that their compatibility with Macintosh computers running Mac OS X is unknown).
Timing considerations
To ensure a correct visual timing, it is critically important to properly select the display technology, frame rate and resolution based on your experimental requirements. For example, if your experiment requires the presentation of very brief stimuli it is advisable to use a CRT display that supports a high frame rate (eg: 120 Hz or above) rather than a LCD display with a 60 Hz frame rate. However, a too high frame rate or too high resolution may put a lot of stress on the experimental setup since they directly affect the required video bandwidth (Bandwidth = Frame Rate x Resolution) and video memory amount to store the stimuli sequence. A too high video bandwidth can lead to unreliable timing if the experimental setup cannot keep up with the workload: while a high-end computer system (i.e. with a fast processor and a modern graphics card) would have no problem dealing with such high requirements, a lower-end system may simply not work reliably. Limiting the frame rate and the resolution to the minimal requirements may however help solve timing issues.
Frame rate and resolution are specified either by the operating system or applications that run in full screen mode. Note that, although the choice of frame rate and resolution is mainly limited by the display capabilities, the graphics card and the operating system can further limit this selection for various reasons. However, it is important to always select a resolution and a frame rate that are the most compatible with the stimulation display to minimize spatial or temporal visual artifacts (see differences between CRT and LCD displays above).
Recommendations
In any case, CRT are recommended over LCD displays because they are less subjects to artifacts and better characterized in terms of spatial, temporal and chromatic properties. If the use of a LCD display is the only viable solution, it is critical to take great care in choosing the specific LCD model to ensure it does not introduce artifacts that would be detrimental to the experimental results (see the X-bit’s Guide for Contemporary LCD Monitor Parameters and Characteristics).
While one expects LCD technology to become more "vision research"-friendly over time, there are other display technologies that may become viable alternatives to LCD in the near future (Plasma, OLED, DLP, LCoS): this section will be updated accordingly to account for display advances relevant to vision research, including links to the development of new display technologies (eg: Laser video display, SED, FED).
Resolution & Frame Rate
Psykinematix supports any frame rate and resolution available through the Display panel under Mac OS X Preferences. The default display configuration (display, resolution and frame rate) to be used for newly-created experiments are specified through the Display Preferences:
The display configuration can be customized on an experiment basis from the Display Settings available from the Experiment properties panel:
As noted above, it may be that not all valid resolutions appear in the Resolution pop-up menu. This is because Mac OS X may not always give access to all of them, in particular those resolutions with a high refresh rate. Fortunately there are third-party programs which allow you to access the additional refresh rates and resolutions supported by your monitor:
SwitchResX (http://www.madrau.com/html/SRX/About.html)
DisplayConfigX (http://www.3dexpress.de/)
Psykinematix automatically switches to the specified resolution and frame rate when running a session in full screen mode, and restores the original display settings when the session ends. However, before running the session Psykinematix can measure the frame rate to check that it is similar to the specified frame rate (see options in the Timing Preferences). This also ensures the CPU usage is not overloaded before starting an experimental session.
How accurate is Psykinematix timing?
Psykinematix uses the OpenGL rendering engine to generate, present and synchronize the visual stimuli with each display refresh. To provide maximum timing precision in visual stimulus onset, offset and duration, Psykinematix takes advantage of the double-buffering and vertical synchronization capabilities of the graphics card.
Since the display constantly redraws the visible video page at a specified frame rate, it could be difficult to make changes to the video page without showing the partial results before a complex graphics operation is complete. Double-buffering provides a way to ensure that the whole stimulus is drawn on the same frame irrespectively of the timing constraints, hence removing visible artifacts from the drawing process that would occur otherwise: the stimulus is written into a back buffer, while the front buffer contains the stimulus currently shown on the display. Vertical synchronization (sometimes called vertical sync or V-sync) is then critical to prevent flickering and tearing in the display when swapping the back and front buffers: the content of the back buffer becomes the content of the front buffer during the vertical blank of the display, and appears all at once to the viewer without artifacts.
The figure below provides a schematic description of Psykinematix display cycle, that is the workload between 2 consecutive frame refreshes and repeated for the whole duration of the stimulation interval. More precisely during every display cycle Psykinematix:
- updates the OpenGL back buffer in preparation of the next frame presentation,
- swaps the front and back buffers in sync with the vertical blanking (VBL),
- sends to external devices any information that requires to be in sync with the onset of the last frame,
- and finally scans the input devices to record responses or wait for input trigger.
Video timing
To maintain a proper timing, it is critical that this cycle is repeated at regular intervals in sync with the refresh rate of the display. As shown in this flow diagram, Psykinematix explicitly controls the buffers swap between each frame presentation (i.e., at frame rate) to ensure that each presented frame is updated with all visual stimuli that need to be shown during the frame interval. To achieve a correct visual timing, each frame must be fully updated between each inter-frame interval, so the maximum duration for drawing all visual stimuli in the back buffer is limited to the inter-frame interval (eg: 16.7 ms for a frame rate of 60 Hz). In other words, the duration of Psykinematix display cycle should not exceed the expected frame duration. This would also ensure that each frame in the front buffer is presented only for this expected duration.
Because Psykinematix runs as a high-priority process, precomputes all necessary stimuli as OpenGL textures preloaded in VRAM, and synchronizes the presentation of each frame to the vertical refresh, Psykinematix can ensure sub-millisecond precision in terms of video timing when its display cycle is not overloaded. Note that the accuracy in video timing depends actually on the input lag introduced by the display (which could be significant for a LCD display, see previous section).
Response timing
Measurements of response timing are always tainted with some uncertainty resulting from some systematic delay (accuracy) and random variation (precision). The sources of timing error in input device responses are numerous (Li et al): mechanical lag, debouncing due to mechanical imperfections, multiple buttons scanning, periodic polling (detection of state changes), event handling by the operating system and the experimental software. For example the timing uncertainty of a standard USB keyboard may include some unknown delays due to mechanical lag, a systematic 20 ms debouncing delay, a 5-20 ms timing bias when scanning the keyboard matrix for multiple keys, a 8 ms from polling (independent of the host computer, operating system or experimental software), and some additional delays due to the operating system and experimental software (in particular those, like in Psykinematix, that explicitly control the refresh of each video frame). Thus the various sources of timing errors in keyboard responses could add up to between 20 and 70 ms uncertainty, large enough to potentially render undetectable the differences between human reaction times measured for multiple experimental conditions.
Because Psykinematix systematically scans the input devices in its display cycle, it also introduces some timing uncertainty similar to the error introduced by a periodic polling. However the total contribution of Psykinematix to the response timing uncertainty is not more than the duration of a single video frame (eg: 16.7 ms for a refresh rate of 60 Hz) and independent of the type of input device.
Under Mac OS X, USB-HID devices like keyboards and mice are polled at a sampling frequency of about 100 - 125 Hz. This means that the operating system only updates mouse and keyboard states every 8-10 milliseconds and therefore that measured response times are quantized with at least that granularity. To the Psykinematix and Mac OS X combined timing uncertainty should be added the device specific timing error from the various sources described above. Because this may lead to large variations and biases in response time measurements in the order of several tens of milliseconds (and Psykinematix not being necessarily the largest source of uncertainty), one has to be very cautious when using normal keyboards or mice for reaction time measurements: although keyboard and mouse are frequently used in measuring response times, their accuracy and precision can be quite low and inadequate for the experimental requirements.
In order to reduce timing uncertainty due to the operating system and experimental software and obtain more accurate measurements, a dedicated response box device (with its own clock and microprocessor) should be used instead to record response times independent of the computer activity, eg: the Cedrus RB Series Response Pads, the ioLab Response Box or the Response Time Box (note that Psykinematix does not support these devices out of the box yet).
Timing warnings
The update of the back buffer and the scanning of input devices are the most time-consuming operations while the buffers swap and output trigger operations are comparatively quasi-instantaneous. If the combined time to update the back buffer and scan the input devices is longer than the cycle period, the buffer swap is delayed to the next frame period and the timing of the current frame is missed. Such overloading of the display cycle results in an unreliable timing. These "missed frames" can specifically occur when:
- too many textures have to be drawn into the OpenGL back buffer in preparation of the next frame presentation,
- scanning the input devices takes too long,
- other programs competes with Psykinematix in terms of CPU or memory usage.
Fortunately, Psykinematix detects and reports the number of missed frames after each session as well as some statistics about the frame duration. Here is an example of session report:
Note that this timing report is only available when running a session in full screen mode during which the buffers swap is under the control of Psykinematix. It is not reported when running a session in window mode (for demo or testing purpose): timing cannot be reliable anymore because the buffers swap is performed by the operating system and only after combining the various windows that compose the full screen.
How can the timing be improved?
As stated above, correct timing depends on the workload in each display cycle, so it is important to minimize this workload to prevent missed frames. The following recommendations can help reduce the occurrence of missed frames from the least to the most restrictive:
- only the necessary input devices should be specified under the Input/Output tab of the Experiment properties. If no special device is in use, there is no need to add any additional device in the input/output table: the default devices indicated in the Response Devices Preferences are automatically added. If no default device is specified in the preferences, then the main keyboard will be automatically available as single input device (it is used for detecting ESC key press anyway).
- there are a few timing options available in the Timing Preferences:
- interruptions from other processes can be minimized by suspending time-consuming applications running in the background,
- the video timing can be checked before starting a session for a specified duration (it is a good idea to have it always checked for at least 1 or 2 seconds),
- the priority of the input/output operations relative to stimulus update during a Psykinematix display cycle can be customized to better fit the experimental requirements,
- the required precision for each frame duration can be increased up to 10% if the experimental setup shows some intrinsic high variability in its timing.
- if the frame rate is not a critical factor in the experimental protocol, decreasing the frame rate should improve the video timing,
- Reduction of the stimuli complexity should only be done as the last resort (eg: by either using smaller stimulus size or decreasing the number of stimuli if too many of them need to be pasted in the frame buffer). Note that decreasing the viewing distance is a “free” way to reduce the size of the stimuli textures!
As emphasized earlier in this chapter, the timing accuracy depends on other factors than Psykinematix per se. If timing is a critical factor in your experiments, you should first try to evaluate the intrinsic timing variability of your experimental setup (computer, display and input/output devices) and if possible independently of the presentation software you are using. There are some commercially available systems dedicated for this purpose, the Black Box ToolKit from example.
References:
Bach M., Meigen T. & Strasburger H. (1997) Raster-scan cathode ray tubes for vision research - limits of resolution in space, time and intensity, and some solutions. Spatial Vision, 10(4):403-414 (HTML Link)
Brainard, D.H., Pelli D.G. & Robson T. (2002). Display characterization. In the Encyclopedia of Imaging Science and Technology. J. Hornak (ed.), Wiley. 172-188 (PDF Link)
Li X., Liang Z, Kleiner M., Lu Z. (B104) RTbox: A Device for Highly Accurate Response Time Measurements. (HTML, PDF links)
Pelli D.G. (1997) Pixel independence: measuring spatial interactions on a CRT display. Spatial Vision, 10(4):443-446. (PDF Link)