AAWSAP DIRD, Technological Approaches to Controlling External Devices, March 2010

UNCLASSIFIED/f 1"8rt 8Ffl@ltltl ~!HI 8HL\f 23 March 2010 !COD: 1 December 2009 DIA-08-1003-012 Defense Intelligence Reference Document Acquisition Threat Support Technological Approaches to Controlling External Devices in the Absence of Limb-Operated Interfaces UNCLASSIFIED//f8R 8FFI&I.t.k Wliili Qtlb¥ UNCLASSIFIED//F8R 8FFl6i,t.k Miili Ot!P X Technological Approaches to Controlling External Devices in the Absence of Limb-Operated Interfaces Prepared by: Acquisition Support Division (DW0-3) Defense Warning Office Directorate for Analysis Defense Intelligence Agency Author: AAP Person 73 Administrative Note COPYRIGHT WARN ING: Further dissemination of the photographs in this publication is not authorized. This product is one in a series of advanced technology reports produced in FY 2009 under the Defense Intelligence Agency, Defense Warning Office's Advanced Aerospace Weapon System Applications (AAWSA) Program. Comments or questions pertaining to this document should be addressed to !AAP Person 1 l AAWSA Program Manager, Defense Intelligence Agency, ATTN: CLAR/DWO-3, Bldg 6000, Washington, DC 20340-5100. UNCLASSIFIED//FQA &FFIGIAk Wiili ,u11a¥ ii UNCLASSIFIED//FOlil OFFI&I.t.k WIiii 8Plklf Contents Introduction ............................................................................................................v Direct Neural Signals.............................................................................................. 1 Indirect Neuronal Signals - The BOLD Effect.......................................................... 3 Control of External Devices .................................................................................... 4 Noninvasive Technologies ...................................................................................... 6 EEG ..................................................................................................................... 7 MEG .................................................................................................................... 8 EMG .................................................................................................................... 9 MRI and fMRI ..................................................................................................... 9 NIRS.................................................................................................................... 10 Invasive Technologies.......................................................................................... 10 Open-Loop Direct Cortical Array Algorithm Modeling ....................................... 11 Closed-Loop Peripheral Arrays Utilizing Visual Feedback ................................. 13 MEMS, ECoG, and PSoC Circuitry ...................................................................... 14 Chronic Neural Implants and fMRI ................................................................... 16 New Electrode Designs..................................................................................... 18 Hybrid Neuro-Robotic Systems for True Closed-Loop BMI ................................ 20 Trials Using Human Subjects............................................................................ 22 Optical Stimulation of Action Potentials ........................................................... 23 Discussion ............................................................................................................ 23 Noninvasive Electrical Devices ......................................................................... 24 Noninvasive BOLD-Based devices..................................................................... 25 Noninvasive Magnetic Devices ......................................................................... 26 Invasive Technologies - General, Optical, and Ex-Vivo Engineering................. 26 Implantable Chips, Ladders, and Arrays ........................................................... 27 UNCLASSIFIED//rOR: orr1e1J1ct U91!! OHL¥ iii UNCLASSIFIED/,'FOR: 8ffl@IJllt t:191!! l>flt I Conclusions .......................................................................................................... 27 Figures Figure 1. Simplified Rendering of a Neuron ............................................................ 1 Figure 4. Experimental Overview of Brain-Controlled Robot in a Closed Loop Figure 9. Image Distortion and Custom Microwire Electrode Assembly to Figure 11. T2 Value Analysis Summary of T2 Values in All the Image Slices That Figure 2. The General Layout of a Closed-Loop Control Interface........................... 5 Figure 3. Two Commercially Available EEG Sensors................................................ 8 With Visual Feedback Experiment ......................................................... 13 Figure s. Schematic of the Neurochip Functional Blocks ...................................... 15 Figure 6. Implant Location ................................................................................... 16 Figure 7. Histology and Electrode Tracks ............................................................. 17 Figure 8. The Michigan Electrodes ........................................................................ 18 Improve It............................................................................................. 19 Figure 10. Example of T2 Variability..................................................................... 19 Spanned the Electrode Arrays in All Animals ....................................... 20 Figure 12. A Hybrid Neuro-Robotic System .......................................................... 21 Figure 13. Experimental Arrangements ................................................................ 21 UNCLASSIFIED//FOil Offl@IJllL YSIE 8Htlf iv UNCLASSIFIED//P(Ht err1e1s1tt l!ISI!!! 9HL'I Technological Approaches to Controlling External Devices in the Absence of Limb-Operated Interfaces Introduction Since the advent of modern interactive control of computers, technologies have been sought to directly connect the biological and the physical to form a seamless entity. While science fiction explores the possibilities of shared consciousness between brains and mainframes, real-life scientists work toward an equally fantastic but more pedestrian goal of eliminating required electromechanical human-machine interfaces (HMis). Such technologies promise integration of thought and computer-controlled action, without the need for a limb-operated device to translate intent from physiological networks to physical circuits. This paper first briefly reviews underlying neural structure, function, and activity to provide background on the type of signals and the scale of temporal changes that arise from conscious control within the nervous system. A short primer on brain-machine interfaces (BMis) is included at the end of the background material. This is followed by a survey of the state-of-the-art detection and stimulation technologies available utilizing noninvasive and invasive methods, including several studies that exemplify research paths currently being undertaken. Discussion brings together elements of the technology survey, application limitations, a timeline for useful commercial deployment, and predicted future research directions. Two technology research paths are highlighted as the most probable to produce functionally useful devices (defined as nearly equal or even superior to the control capabilities of current HMis) in the near and far term. The technology endpoint this paper seeks is thought-based operation of remote machinery during normal human activities without mechanical device interaction-that is, control of external devices without the need to go to a specified location, such as a shielded room; without the need to remain perfectly motionless to reduce signal noise; and without the need for interaction with a normal electro-mechanical device, such as an i-Phone or other handheld device with buttons and trackballs. Data transfer rates are sought to exceed 5-10 bits/second to be useful for operation of complex devices; this rate range and above is referred to as high-bandwidth BMis. Response time for a 1-of-N selection of commands is targeted at 300 milliseconds or less. These are the operational parameters assumed for final implementation unless some limitation or supercapability is described. The paper concludes that noninvasive electrical monitoring of neural activity, primarily reading combined action of muscles and neurons, is the most promising commercial technology in the near term. Inherent limitations in the noninvasive electrical approach require future development of different research paths. The paper argues, mainly by eliminating other approaches, that the most probable technology in the long term involves invasive single­ neuron-based direct cortical connections to form a network of high-bandwidth duplex communication pathways. Currently the most promising technologies UNCLASSIFIED//FOR. OFFIGIAk Uii QPlk¥ V UNCLASSIFIED//P91t 9Ffl@IAL YSIE 8HLY for robust construction of such an interface are optical stimulation, gating, and sensory devices, and chip-based electrode arrays that have been encased in ex-vivo-engineered neural tissue. UNCLASSIFIED/ /FOR. orrlEIAL U.!~ Gilt I vi UNCLASSIFIED/ /POR: err1e1At l:191!! er•t I Direct Neural Signals The human nervous system has two classes of cells, neurons and glia. Based on all research to date, it is believed that signals within the network of neurons constitute the whole of information processing, with glial cells playing a purely supporting role. This neural doctrine has dominated research in BMI until recently and still constitutes the only major research path in direct technologies. Furthermore, all technologies directly measuring human neuronal action rely on detecting or influencing electrical activity of these cells; no current in situ research selectively affects neurotransmitter activity between local cells for the purpose of information exchange . Therefore, the focus for the foreseeable future will be on the electrical activity of neurons as the primary target of BMI. Neurons consist of four parts: axon, dendrites, cell body or soma, and pre-synaptic terminals. Electrical information is transmitted to the neuron through the dendrites, proceeds through the cell body, and leaves the cell through the axon at one or more pre-synaptic terminals. Neurons have one axon and from one to tens of thousands of dendrites. Figure 1. Simplified Rendering of a Neuron. The arrows indicate t he direction in which signals are conveyed. The single axon conducts signals away from the cell body, while the multiple dendrites receive signals from the axons of other neurons. The nerve terminals end on the dendrites or cell body of other neurons or on other cell types, such as muscle or gland cells. (Reference 1) Chemical details of how the action potentials travel through the cell or are transmitted across the synapse are not important to the current treatise, other than the distinction that in these biologically based electrical networks, ions of sodium, potassium, and chlorine move through the cell membranes perpendicular to the propagation of the action potential down the axon. This allows information to be transmitted faster than ions could flow down the axon. The propagation of information is similar to a wave traveling down a garden hose: quickly move one end of the hose back and forth with sufficient force, and a wave will travel to the other end of the hose; however, any part of the hose structure has only moved (nominally) perpendicular to the direction of wave propagation. In a similar fashion, ions flow through channels across the axon's cell membrane, changing the local membrane potential and thus propagating the electrical signal down the axon. UNCLASSIFIED/;'FOR OFFI&I.t.k Wliliii 8fslk¥ 1 UNCLASSIFIED/;'FOR OFFl&I.t.k Wliliii &NL¥ The signal transmission down the axon of a neuron is an all-or-nothing process. When the cell body is stimulated above threshold, the axon transmits the same action potential at the same speed and in the same direction, regardless of the extent above threshold or duration of the input. Action potentials have durations of 1-10 milliseconds. Input signals can result in transmission of multiple action potentials, and thus the frequency and number of neuronal firings do vary with the input. Neurons require some time to reset between firings, nominally the duration of the pulse for that axon, yielding a typical maximum firing rate of between 100 hertz (Hz) and 1 kilohertz (kHz). It is instructional at this point to contrast this mechanism with propagation of signals through a physical electrical circuit, the planned external portion of our BMI. In a copper wire, electrons carry the signal. Electrons drift along the signal path, but the signal itself moves as a compression wave rather than a transverse wave as in the biological system. Going back to our garden hose example, consider the hose now filled with small marbles: inserting a marble at one end will move each marble in the hose just a little, but very rapidly the last marble in line will pop out of the far end of the hose. As with the biological system, the signal is propagated to the far end of the hose by local actors rather than physical motion of a single ion or electron moving the whole distance. The underlying physics governing the signal transmission makes metallic and semimetallic circuits about a million times faster than the biological system. This difference in the carriers and underlying mechanisms of signal transmission between biological and physical circuits has so far prevented the invention of a direct connection between the two disparate systems. Instead, both noninvasive and invasive direct-detection technologies rely on placing physical sensors or transmitters in close proximity to the neurons of interest and utilizing classical electrodynamics to govern signal jump between the systems. Additionally, given the maximum typical firing rate for neurons of 1 kHz, sampling of action potentials at a few kHz will be fast enough to detect firing of any individual neuron, though super-sampling above 10 kHz can be used to reduce noise. Higher sampling frequencies also may be required if multiple neurons are monitored and quantitative information about their relative firing sequence is desired . Frequencies of a 1 kHz or below are sufficient to stimulate action potentials, and again, higher system frequency may be required for multiple neuron sequential stimulation. Finally, higher frequencies may be required if monitoring or stimulation of some aspect of signal transmission other than action potentials is sought, such as monitoring single ion channels. 1 There is currently no evidence that transduction frequencies above 100 kHz have any advantage in BMis, thus the main challenge is the connection dynamics, since even this ultra-maximum frequency is easily attainable with current electronic manufacturing technology. The human brain doesn't process information as a traditional computer does. Information is moved around through pathways and at certain neurons it is allowed or not allowed to pass based on excitory or inhibitory dendritic signals arriving before triggering of action potentials. Local groups of neurons can act nearly coherently, for example in volition of motor action like a hand movement. Detecting such coherent firing at nodes around the brain is robust both noninvasively and invasively, though 1 Technologies that monitor single ion channels on a neuronal membrane are important for research on neural function and neurotransmitter action. However, these devices are ultra-sensitive to physical motion, and have so far not proven useful in large scale information transfer required in BMis. UNCLASSIFIED//fOll OFFl@IAl W&lii OPU..¥ 2 UNCLASSIFIED//FOR OFFl&IAk le.ISi: OPtklf noninvasive techniques currently cannot resolve firing sequences of individual neurons within such groups. Living neurons in an active tissue fire in the rest state. Changes in the frequency of these firings imply that a given neuron is currently involved in the processing of information. Bulk changes in local field potential oscillations imply that several neurons are active. This is the signal seen in the noninvasive direct-measurement techniques of electroencephalography (EEG) and magnetoencephalography (MEG). Movement as simple as an eye blink involves signal communication through a million neurons. Several locales in the nervous system thus offer the possibility of placement of an invasive monitoring device for detecting electrical activity associated with an eye blink or other signal of interest. Many other factors will influence the placement decision, but considering only catching the signal as an action potential, using any neuron along the complete pathway is equally as good, assuming detection of individual neuronal firing can be accomplished. Considering cortical placement of detection devices, the most common area of activity under study is the outer covering of gray matter of the cerebral cortex, the neocortex. The neocortex is about 2. 5 mm thick in humans and follows the ridges and fissures of the brain (gyri and sucli, respectively, or gyrus and sulcus, if singular). The neocortex is roughly divided into six layers and different cortical probes can concentrate on activity in different layers, or just measure the combined activity on the surface of the cortex, or the combined activity that is detectable noninvasively through the layers of tissue and bone of the cranium. Detecting single firings of individual neurons is a difficult process because the signals are weak to start with, and not isolated from the rest of the electrical activity within the brain. Large groups of coherent neurons, perhaps a few thousand to tens of thousands all firing at once in relation to an external event, are the most studied of single firing signals. 2 One of the most studied signals is the so-called P300 which occurs in many cognitive tasks (References 2-4). P300 is a term used to indicate a pulse signal from a large group of neurons that appears about 300 milliseconds after a stimulus event. Peaks that occur between 200 and 400 milliseconds are commonly grouped into the P300 category. The "P" stands for a positive measured voltage, while "N" signals are negative. 3 The P300 signal has been shown to be influenced by top-down executive function and is therefore a prime candidate for a trainable interface (Reference 5). Earlier signals, like PS0 and Nl00, are of a greater interest for differential diagnosis of pathology in current research. The preceding discussion is greatly simplified version of the electrical dynamics of neuronal firing and does not include differences between axon and dendrite signals, or the transmission of signals across the synapse. Complete discussion of underlying electrical signals in the nervous system can be found in (References 6, 7). Indirect Neuronal Signals - The BOLD Effect 2 Single firing signals here mean a single peak of combined electrical activity relative to an event. This is not necessarily the same as single firings of each neuron. 3 The literature is not consistent with plotting P signals up, and N signals down; however, P peaks are always in the opposite direction of N peaks. UNCLASSIFIED//POil OPPl@IAL liSI!! 8HLY 3 UNCLASSIFIED//EOR OlifilEIAk W&li (UtLY The brain activity mentioned above is a complex chain of ionic motion within the central nervous system. Ionic production, release, and movement within and between cells all consume energy. This energy is supplied by the conversion of blood-borne oxyhemoglobin to deoxyhemoglobin. The rate of oxygen consumption in a localized volume varies based on local neural activity. The circulatory system compensates for changes in energy demand by increasing or decreasing both the flow rate and volume of blood, regionally and locally. Local energy demand, expressed in the capillary beds, will alter the rate at which oxygen is metabolized, called the cerebral rate of oxygen metabolism, abbreviated CMRO2. When brai n activity increases in a region, the circulatory response, called the hemodynamic response, will be increases in flow and volume, while the local areas increase CMRO2. The hemodynamic response consistently provides an excess of oxygen over what is required, and this results in some oxyhemoglobin traveling through the capillary bed and local venous structure without being converted into deoxyhemoglobin. Oxy- and deoxyhemoglobin have different magnetic susceptibilities, and different infrared spectra. The hemodynamic response, by changing the net ratio of oxy- to deoxyhemoglobin in the local venous structure, thus changes the local magnetic susceptibility and local infrared resonance spectra around focused brain activity. This complex chain reaction is called the Blood Oxygen Level Dependent, or BOLD effect (Reference 8). The BOLD effect leads to a method to indirectly measure the local brain activity by monitoring the hemodynamic response using Magnetic Resonance Imaging (MRI) or Near Infrared Spectroscopy (NIRS). The BOLD response to any event peaks about 4-6 seconds a~er the event occurs, limiting the applications for which monitoring these signals and their associated delay may be useful, given the BMI operating parameters defined above Furthermore, person to person variation in distributed signals show significant differences in regions activated (Reference 9), though there is evidence that these inter-subject variations are stable intra-subject over time (Reference 10). Control of External Devices In general, there are two types of BMI design called open-loop and closed-loop. Open loop designs, those either detecting activity or providing stimulation, are important for BMI research and application other than the practical control of an external device. Closed-loop systems include both output signals from the brain to effect a change in the state of a device, and a stimulation channel, which can be used as feedback. In some system designs, the feedback can be via visual cues such as watching a display (Reference 11). Open-loop systems are important for BMI research and have application in sense replacement such as artificial hearing, or when extensive output signal processing is combined with external device state feedback (see remote robot operation below). For learning and fine, efficient control of external devices without significant external processing, closed-loop systems are typically needed . Several components constitute a general BMI system for control of an external device (Figure 2). As mentioned in the previous section, the interfaces where the information signal makes the jump between biological and physical pathways are the most challenging part of the system. These connections can be placed to stimulate or read peripheral or cerebral neurons. The processing step could be as simple as amplification UNCLASSIFIED//FOR OFFl@IAL Y!H! 8HL1/ 4 UNCLASSIFIED/ /FOR: 8Ffl@IJltt t:191!! er•t I of a spiking train to trigger a switch, or as complex as decoding signal from noise utilizing a 300-channel EEG cap. ~-------------------------, : Additional sensory : I ---1 -------------------' ~ Brain Output interface ,----1 Input interface Output signal processing Components of a Closed-loop BMI Control System Input signal processing External Device Figure 2. The General Layout of a Closed-Loop Control Interface. The input and output interfaces can attach to read or affect activity of cerebral or peripheral neurons. Additional sensory feedback channels such as visual displays, audio tones, or haptic feedback are possible. In some system designs, especially noninvasive types, these sensory feedback channels replace stimulation inputs. In an open-loop interface, either the input or output signal paths are absent. One consideration in developing a BMI control system is the utility of the application: does it make sense to directly tie these controls to neural activity? The bulk of research in direct neural interfaces is toward the goal of restoring mechanical capabilities to those individuals who have either lost limbs or lost control of limbs through central nervous system injury or disease. In these cases it is of obvious utility to produce a system that moves a cursor across screen to select an item over the course of several seconds; however, for a healthy individual, it is far more practical to execute a hand movement to control a mechanical device4 to achieve the same goal. An exception to consider is the case of applications where the subject's hands or feet are already occupied with other essential tasks in the overall application. A research question for any such application is whether training a subject to utilize a BMI is advantageous over a more complex mechanical interface. Furthermore, whether training on a new BMI affects learning/retaining proficiency with similar BMis is an additional concern. Moreover, for any BMI, one can quantitatively describe the information transfer in controlling the magnitude of human motor action (Reference 12). Since the advent of digital control of external devices, this information is expressed in equivalent bits per 4 Mechanical device as used here represents any limb-operated interface, such as a mouse or touchscreen. UNCLASSIFIED//FOR OFFI&IAk W&E 8Ptk¥ 5 UNCLASSIFIED//POil 8ffl61Al W&li 0NL¥ second transferred. Higher bit rates equate to more efficient interfaces. Baseline examples of performance include finger pointing, which can convey 14 bits/s of information (Reference 13), operation of a mouse about 8 bits/s (Reference 14), while stylus tapping a soft QWERTY keyboard on a PDA has a lower rate, around 5 bits/s (Reference 15). Once these questions are researched and answered for a given application, the design choice should be made whether a new mechanical interface, or an optical interface that tracking detailed motion of the eyes, body and limbs, may be the optimum solution rather than a BMI. 5 Another consideration in developing a BMI control system is the cognitive limit of supervisory control of external devices. The current state of the art for human interfaces involves volition of control, meaning that there is some executive supervision, typically involving the pre-frontal cortex, active during the control process. Recent studies relating to control of semiautonomous vehicles show that there are significant limits on the number of supervisory activities that can be undertaken simultaneously (Reference 16). In response to these limits, one can consider utilization of nonsupervised brain networks in control of external devices, similar to those brain systems that initiate very complex muscle coordination activity such as martial arts or playing a musical instrument, or systems like those that are responsible for autonomic function. Application from such research is long term, but there are current studies that show BMI training of at least one spinal cord patient exhibited differential activity in the cerebellum, owing to extraordinary plasticity (Reference 17). It is therefore not out of the realm of possibility that sufficient input and output interfaces that include neural pathways through the cerebellum could result in over-learning a BMI application to the point that supervisory control is developed and thus significant cognitive effort is not required. Noninvasive Technologies There are four noninvasive technologies currently in widespread use in monitoring brain activity. The two direct-measure technologies involve detecting surface currents from the relative voltage changes at or just below the scalp, electroencephalography (EEG), and detection of near scalp magnetic fields associated with neural pathway current flow, magnetoencephalography (MEG). The remaining two indirect measures involve monitoring BOLD response through rapid successions of whole brain MRI (functional MRI or fMRI), and near-infrared spectroscopy (NIRS). 6 The primary technology used for BMI is also the oldest. EEG was first described in 1929 (Reference 18) and now exists in several derivative forms. Traditional EEG uses electrodes at the surface of the scalp to measure and amplify potential differences between points above the cortical surface and a fixed reference such as the average reading from the earlobes. Traditional EEG data is analyzed by breaking up the power spectrum into several bands between 0.5 and 100 Hz. A derivative form of EEG developed about 10 years later is called event-related potentials, or ERP. In ERP, scalp 5 There is a valid argument to be made that optical tracking and analysis, especially of the eyes, is a BMI technology as defined in the current treatise. It is excluded as one merely by fiat. 6 NIRS is occasionally referred to as functional NIRS or fNIRS. NIRS using multiple sources to produce 3D images of internal changes in blood flow is occasionally called diffuse optical tomography (DOT). DOT is a more general technology that can also refer methods such as to visible-light laser excitation of tissue. UNCLASSIFIED//FOil OFFl61Al WSE OHlY 6 UNCLASSIFIED/ j POI\ OFFICIAL l:ISE 8Ptllf data is averaged over several electrodes time-locked to a stimulus (Reference 19). Both the previous methods record summed electrical activity of nominally 50,000 local neurons, thus large coherent group spiking activity7 is required to produce appreciable signal. EEG EEG-based BMI systems use pattern recognition among the several electrodes to transmit information. In closed-loop systems, audio or visual feedback is utilized. Training on the system and adaptation by the processing software can increase target detection efficiency, though some recent work shows that systems can be produced that nearly work right out of the box. Blankertz demonstrated a system that required only 20 minutes of training on naNe subjects (Reference 20). Development is partially being driven by the need to provide a means of communication or action on the environment to patients that have lost control of their body. Much of this clinically-orientated research has focused on 'locked-in' patients, who suffer from total paralysis following brainstem stroke or degenerative diseases such as amyotrophic lateral sclerosis (ALS) (Reference 21). The goal has been to extract control signals either from surface EEG signals or from electrodes implanted near or within the cerebral cortex: ECoG (see next section on invasive technologies for details on ECoG). ALS deteriorates peripheral motor function before progression to cognitive areas. This greatly reduces a primary source of bioelectric noise, cranial and facial muscle action, and thus provides a reduced signal processing problem to decode neural signals. Successful communication has been established via EEG BMI in several studies based on both spiking activity and P300 signals (References 22-25). The response time to execute a command using these systems is measured in seconds. The results from ALS patients represent a best-case scenario for what could be accomplished using EEG-only sensors in a normal, healthy human who would have significant muscle noise to sift through. The number of electrodes and the time-consuming application of conducting gel make EEG arrays bulky and slow to put on. An improvement would be a reduction in the number of electrodes, typically a few dozen for traditional EEG, and electrodes that could operate without conducting gel. Commercial versions of dry-electrode devices are being released on the market from companies such as Neurosky, OCZ Technology, and Emotiv.8 The Neurosky and Emotive systems claim proprietary processing algorithms, lack peer-reviewed literature supporting their claims, and even discuss using facial muscle movement to send signals (References 26, 27), raising doubt about whether neural signals are being measured at all. The NIA, for Neural Impulse Actuator, from OCZ clearly states that the electrodes pick up a combination of EEG, EMG, and EOG,9 and that the algorithm is only interpreting the net signal patterns instead of trying to sort out the EEG component. 7 Spiking activity is the term used for recognition of action potentials. Spikes are fast and easy to recognize with electronic triggering circuits, while more complex waveforms require additional processing . 8 Websites www.emotiv.com, www.ocztechnology.com, and www.neurosky .com, last accessed 12 May 2009. 9 The electrco-myogram (EMG) reads signals from muscle activity, in this case facial muscles. The electro­ oculogram (EOG) measures electrical signals arising from muscles associated with the eyes. 7 UNCLASSIFIED/fFOR OFFI&I.t.L W&E 8Pttlf UNCLASSIFIED//FOR [email protected] WSI: Ortklf Commercial EEG sensors Mindset, by NeuroSky nia, by OCZ Technology Figure 3. Two Commercially Available EEG Sensors. The Mindset utilizes two sensors, one in the right earphone and one for placement on the forehead (what looks like a microphone arm is to be pressed against the forehead). The nia records signals from three sensors on the forehead . In a documented and peer-reviewed study, Popescu and colleagues have shown a system that uses 6 dry electrodes and is 90 percent accurate in operation of a 1-D cursor by untrained subjects. Additionally, a measurement of error rates as a function of number of electrodes show that an increase to 12 electrodes could drop the error rate to around 5 percent, but that additional electrodes much beyond a dozen do not significantly improve the accuracy of their algorithm (Reference 28). MEG MEG-based BMI systems have been shown to be feasible utilizing a subject imagining limb movements for binary decisions (References 29, 30). Future work could improve the methodology to parallel that achieved with EEG; however, there are significant technological hurdles to a field deployment of MEG. The magnetic fields from neural activity are detected with very sensitive devices called superconducting quantum interference devices, or SQUIDs. These detectors are sensitive to the neural activity induced 100 fT (femtoTesla) changes near the scalp. SQUIDs only operate at very low temperatures thus requiring a cryogenic system as well significant detection and amplification electronics. The availability of a cold sink and high vacuum, such as in a space-based application, could reduce the support system overhead, but there is still the matter of the weakness of the signals. Even if future detector development like atomic magnetometers would solve the equipment overhead issue for Earthbound application, the fact that 100 fT is about 100 million times smaller than the Earth's magnetic field will prove an insurmountable barrier to sifting signal from noise in anything but a heavily-shielded, metal-free environment. Attempts have been made to use High-Tc superconductors as a shielding material with some success (Reference 31), but significant further development is needed. UNCLASSIFIED//FOR OFFIEiIAk WSIE 8HL1/ 8 UNCLASSIFIED//FOA OFFIEill.tL l::181: 8HLY EMG Instead of measuring nerve impulses and amplifying them, one can design a system that monitors muscle movement in a hands-free approach to a mechanical interface, essentially using the muscles as a biological amplifier of neural signals. One may close the feedback loop using a heads-up display, monocle, or other visual device, or another traditional sensory feedback. The advantage of such a design is that controls may be actuated through a lightweight wireless system, effectively providing control stations anywhere such a wireless system would work. Such a system could be used to untie a pilot from the cockpit. Recent work has shown that performance of 1-2 bits/s is possible with minimal training, about four times the current performance of a comparison EEG forehead sensor (Reference 14). MRI AND FMRI Magnetic Resonance Imaging (MRI) works by a simple excitation and relaxation of spin states. When molecules containing hydrogen are placed in a strong static magnetic field, a small but detectable number of hydrogen protons align their intrinsic spins along the direction of the external field. An applied radiofrequency (RF) pulse near the proton resonant frequency, 42.6 MHz/Tesla or 128 MHz at 3 Tesla, knocks the spins perpendicular to the field and the relaxation back to ground state releases RF energy in patterns that can be reconstructed to show both composition and distribution of any hydrogen-rich materiel. The resonant frequency is a direct function of the local magnetic field defined by the Larmor relation: c.o = y B; where c.o is the frequency of precession, B is the local magnetic field, and y is a constant of the material, 42.6 MHz/Tesla for bare protons as mentioned above. Small perturbations to the static field will change the resonant frequency. By applying a small gradient to the static field, for example 100 milliTesla/meter along the z-axis, and limiting the bandwidth of the RF excitation signal to 8ro, one may select a slice of the brain perpendicular to the z-axis for excitation to 82. A change in the gradient field will change the position of the excited slice for the next excitation. Similar gradients in the x and y directions can limit the excitation to a single small volume of brain tissue. In current MRis, these gradient fields are produced with electromagnets, and the series of time-dependent imaging gradient manipulations is called the scan sequence. Free hydrogen (H) would produce a resonant signal slightly different than the bare proton due to the local field changes induced by its valance electron. Hydrogen gas (H2) would produce a still different frequency since the local field around each proton is altered by the two shared electrons. Water molecules (H2O) contain two hydrogen atoms and an entirely different "electron shield" than either H or H2 and thus shows still another slightly different resonant frequency. Fat and other lipid molecules, important cell structure building blocks, have long chains of hydrocarbons, and the resulting ensemble of electron screening produces a wide peak that is substantially shifted 10 from that of water. Brain gray matter and white matter have different macroscopic lipid content and are thus able to be differentiated in an MRI scan. Different signals also arise in bone, 1° Frequency detection sensitivities in MRI are very good, and "substantial" here means about 3 parts per million. The frequency shifts caused by imaging gradients ranges in the parts per thousand. UNCLASSIFIED/f FOA OFFIEiIAL l::ISE 8PtLY 9 UNCLASSIFIED//P91t 9Ffl@IAL YSIE 8HLY cerebral-spinal fluid (CSF), and internal tissue structures of various other organs. Unlike x-ray based technologies, MRI scans can be optimized to contrast any of the many parts of the physics signal: total density of protons, water content, lipid content, and even particle motion in advanced techniques involving diffusion or spin labeling. Using such scans sequences that take several minutes, one can construct very high resolution images of gray and white matter structure for comparison with, and also mapping onto a "standard brain" template. In addition to electron screening, macroscopic susceptibility will also change the local response to RF stimulation. The presence of even a small amount of metal, say as small as a hairpin, will greatly distort the reconstructed images. Indeed, the usual effect is a shift of frequency completely outside the sensitivity of the machine RF receiver in what is known as "drop-out." Smaller changes in local susceptibility, like produced in the BOLD effect, are measureable. A series of fast scan sequences, typically collecting an entire brain volume at a resolution of 3 mm3 in 2 seconds, that are calibrated to optimize detection of the BOLD signal will show the dynamics of brain function under the specific internal or applied conditions; this is known as a functional MRI, or simply fMRI. 11 The major advantages of fMRI are unmatched 3D spatial resolution, compared to other noninvasive imaging methods, and complete skull penetration, making it the only imaging modality to unambiguously detect limbic activations important for determining emotionally-laden neuropsychological states. The main disadvantage for BMI is that the BOLD signal is detected several seconds after the neuronal firing takes place, making fMRI inappropriate for many naturalistic applications. A long term prospect, likely in the 20- to 40-year timeframe, is that combined low-field MRI and MEG technology could detect neuronal firing deep in the brain and with high temporal accuracy. Initial experiments indicate some level of feasibility, but there is substantial development work required in room temperature low field magnetic field detection devices, such as atomic magnetometers, and signal processing algorithms to sift through the substantial electromagnetic background (References 32, 33). NIRS Near-infrared spectroscopy is an additional technology to monitor the BOLD effect noninvasively. Studies have shown it correlates well with the fMRI signal in animal models, although with reduced coverage and lower resolution (Reference 34). This lowered resolution greatly affects the accuracy of the technique, with recent work involving single trials and a decision attaining only 80 percent accuracy (Reference 35). Invasive Technologies The most prolific invasive BMI for use in humans is the cochlear implant (Reference 36), a sensory neuroprosthesis designed to aid in hearing for deaf individuals. This device, under continued development and refinement for more than 30 years, consists of a microphone, sound processor, and a receiver that is attached to an array of 11 Specifically this is T2* Echo-Planar Imaging, also called BOLD EPI, Gradient Echo EPI, or BOLD fMRI. This approach is used in well over 95 percent of published functional studies, though there are more advanced techniques that concentrate on smaller portions of the hemodynamic signal. For example, Spin-Echo EPI will provide a higher localization within the gray matter, but the cost is a loss of 90 percent of the signal amplitude. UNCLASSIFIED//FOR OFFI&Il.tk WSE 8Hk¥ 10 UNCLASSIFIED//FOR 8FFI€iIAL YSE 8,.LV currently 22 electrodes directly implanted into the cochlea of the inner ear. The microphone and sound processor replaces the outer and middle ear, while the electrodes replace the frequency selective hair fibers of the inner ear that normally would transmit electrical signals to the cochlea. The quality of the sound is much less than natural hearing, but the only option for patients who have lost hearing or were born deaf. It has been estimated that 100,000 cochlear implants provide sound to users worldwide (Reference 37). The cochlear implant is an open loop stimulation system that provides complex signals to the nervous system through a peripheral connection. BMI technologies have evolved from assistive technology devices solely targeted for the healthcare industry, into apparatuses designed for intracortical microstimulation to deliver sensory feedback. Such systems allow the simultaneous recording and microstimulation of neuronal and behavioral events. The primary device that has emerged leading this technological transformation is the direct cortical array containing either surface or neocortex penetrating electrodes. The direct cortical array is implanted on the surface of the brain near the functional area of interest, and then the connection is reverse-engineered: individual electrodes are tested to determine if they should be assigned as input, output, or neither. 12 These devices are designed to fuse neural signaling pathways with external machinery into a 3D control system, such as those apparatuses designed to modulate the movement of appendages (e.g. robotic arms), though the signals could be used for control of any complex machine. The control circuits on these systems are intricate and often bulky, and they can be designed with either open- or closed-loop sensory feedback. An additional device is the penetrating cortical electrode. These electrodes contain a ladder of input/output probes and can be selected to transmit or receive signals in any layer of the neocortex. Research is ongoing regarding the optimal layer with which to place an interface. When using penetrating electrodes, tissue response is an additional challenge as these devices frequently cause scarring. Animal studies are in progress to quantify this effect. Technological challenges in development of all types of invasive BMI devices include signal decoding/stimulation algorithms, and localization of brain activation near the implant. Functional MRI (fMRI) is utilized in conjunction with the invasive implants to confirm additional neural network activity during interface tasks. Modifications in the design of electrodes is necessary for experiments involving fMRI (Reference 38). 13 The majority of invasive experiments to date use animal or even tissue models, though a few human trials have been conducted (with the exception of the cochlear implant described above) . OPEN-LOOP DIRECT CORTICAL ARRAY ALGORITHM MODELING In traditional open loop experiments, eye movement systems are utilized to develop a linear model for a physiological system. 14 In such experiments, invasive signals are recorded from the cortex and the motion of the eye is recorded in tandem. The location t i This procedure eliminates the need to directly locate the neuron(s) of interest during implantation. 13 Field potentials at the points of metal electrodes cause concentration of RF energy from the excitation pulses of the MRI. This effect is used in a similar setup for tissue ablation therapy; however, such energy concentration is undesirable in the studies at hand . 1• These techniques may be generally applied to any motor control systems. UNCLASSIFIED/ /FOR. 8fPltlAL U.91!! f>flt I 11 UNCLASSIFIED//P91t 9Ffl@IAL YSIE 8HLY on the cortex to implant cortical arrays is determined from previous experiments in fMRI (References 39, 40). In this way, a map is constructed correlating electrical neural activity with muscle movement. The goal of this exercise is to develop an algorithm that will predict which muscles move based on reading the neural activity alone. The reading of the neural activity for eye movement can then be used to move a device such as a camera lens. There are two main models of fine motor control, one where the cortical motor areas perform all of the control functions and receive all sensory feedback, and a second where the cortical areas direct the function and receive interpreted feedback through sub-cortical or even peripheral networks. For open-loop invasive BMI applications, this is an academic question since peripheral interfaces would receive and send the same signals in both models, and cortical interfaces would blindly adapt external decoding algorithms based on the signals present regardless of model. There have been numerous demonstrations of nonhuman primates controlling robots or graphical cursors in real-time through signals collected from cortical areas that employ open loop experimentation (References 41, 42). Kim, et al. conducted experiments where monkeys are trained on tasks prior to implantation, and then the tasks are repeated multiple times while muscle action and cortical activity are monitored (Reference 42). In these trials, shoulder and elbow torque were measured while the arm itself was constrained in an exoskeleton such that the hand would only move in a plane axial to the monkey's torso. A visual cursor was introduced and projected on a screen above the monkeys hand to follow the 2-D motion from the center starting point to the various task targets, which are also projected on the screen. The shoulder and elbow position recorded the state of flexation of 6 sets of muscle groups, collectively called the musculoskeletal arm model (MAM). Relating the spiking activity from implanted arrays to even this simplified 2-D MAM motion proved quite complex, and no fit correlating the observed movements and neural activity could be obtained with a linear model when kinematic impedance was considered. 15 Even considering the six inputs, the 2-D problem is essentially a computer cursor control and therefore a relatively simple device, fully specified by a Cartesian coordinate system. The ultimate goal of these control systems is to manipulate something much more complex, like an arm, which may have many more degrees of freedom organized in a completely different coordinate system. For these tests a more elaborate "tracking system" may be utilized in teaching a primate to feed itself16 using a directly observed cortically controlled robotic arm. Open-loop control systems have an inherent drawback in cases where cortical activity controls movement directly via an adaptive algorithm. Training the algorithm is the critical part of interface development. The adaptive algorithms use an iterative process to create a brain-to-cursor motion decoding scheme based on how the neurons fire when different targets are presented and therefore rely on previous normal feedback training - the brain knows how to move an arm because it has been moving an arm for most of its life. 15 Impedance to motion is essential for realistic operation of artificial limbs. 16 Food in this experiment is used as a reward. UNCLASSIFIED//F8R 8FFI&I.t.k Y&liii 8,.kY 12 UNCLASSIFIED/ /FOR [email protected] WSI: Ortklf An issue develops when an entirely new device is considered. The form of the movement control algorithm is similar to a population vector in that movement at each time step is determined by a vector sum of the neurons' normalized firing rates multiplied by a set of linear coefficients. Modern neuroscience has proven that tens of thousands, even up to millions of neurons fire in concert to perform even the most mundane of movements. These extended neural pathways developed through full­ duplex closed loop training. It will thus be quite an issue in an open-loop design to develop an algorithm that can decode control signals for a system the brain has never controlled before, say a 12-direction rocket stabilization system. 17 This places another limitation on the open-loop model, even using complex nonlinear fitting, and has driven development of the closed-loop model. CLOSED-LOOP PERIPHERAL ARRAYS UTILIZING VISUAL FEEDBACK In a closed-loop system that provides feedback to a control system, the differences between the two models of movement become apparent. Sensory feedback within the system is used for error correction of fine motor control, such as balance - without fine motor control and error correction for balance, the human body could not stand up. In the model with abstract cortical control, feedback corrections are processed externally to the BMI and sensory feedback to the brain is usually limited to observation. This is a subtle distinction between open-loop with visual feedback and closed-loop abstract control that includes visual feedback. In the former, the visual feedback is intended to show success of a control command being sent by the brain, whereas in the latter, visual feedback is showing success of supervisory function of a semiautonomous mechanical device. eural Recording JAPAN Figure 4. Experimental Overview of Brain-Controlled Robot in a Closed-Loop With Visual Feedback Experiment. After decoding walking-related Information from a monkey's brain activity while walking on a treadmill, these data were relayed from Duke University in USA to the Advanced Telecommunication Research in 17 Twelve directions are the minimum number of positive controls commands to +/- x, y, and z thrust, as well as increasing or decreasing roll, pitch, and yaw. These are six degrees of freedom but as far as a BMI control system is concerned, increasing x and decreasing x are separate commands to decode. UNCLASSIFIED/ /iiiOlil OiiiiiilCI0ls !lili OPII.¥ 13 UNCLASSIFIED//P91t 9FFl@IAL YSIE 8HLY Japan in real time. Using visual feedback to the monkey via live streaming video, the humanoid robot in Japan was shown to execute locomotion-like movements in a similar manner as the monkey. (Reference 43) An example of a closed loop peripheral system is the creation of the humanoid robot called CB-i (Computational Brain Interface) . This system is illustrated in Figure 4. In a recent experiment, CB-i successfully mimicked the physical actions being performed by a monkey that was positioned in a remote location (Reference 43). MEMS, ECOG, AND PSOC CIRCUITRY To address the issue of signal interface between the biological and physical systems, development has focused on hybrid devices that are created by fusing together biomolecules, cells, and other tissues, with innovative micro-size current sensors using Micro Electro Mechanical Systems (MEMS). These systems often employ the use of biomaterials and functional high polymer materials that enable the device to sense information about the human body and the environment with greater speed and sensitivity than conventional metallic sensors. These devices are built from materials and mechanisms that are compatible with the human body, and they are proving to be powerful tools for interfacing between the human body and machines. The emergence of MEMS biotechnology has motivated various strategies to detect electrical signals generated from a stimulated region within the brain into a captured electrical response that is translated to a machine interface. One advantage of these devices is that the fusing of biological to physical circuitry is performed external to the body and the interface connection of the implants is thus biological to biological. This avoids a natural reaction of the nervous system to form scar tissue around penetrating chronic physical probes. 18 Scar tissue leads to degradation of proximal signal transfer over time. These neural-tissue encapsulated chips offer significant advantages over other technologies because of their ability to be rapidly integrated into the biological environment. Noninvasive EEG platforms often utilize a cap that covers the skull and reads the electrical signals generated from the brain on the surface of the skin atop the head. However, a technical limitation in the bandwidth of EEG-based methods often provides low information rates. Currently EEG methods are limited to 20-30 bits/min ( < 0.5 bits/s). This drawback has prompted the use of a more accurate and speedy recording method based on invasive techniques such as the electrocorticogram (ECoG), where electrode arrays are placed on top of the cortex, but electrode penetration into brain tissue is minimal. ECoGs have become synonymous with the pre-surgical monitoring of epileptic seizure foci (Reference 45). Valuable insights can be gained by combining both invasive and noninvasive schemes. For example, combining functional MRI (fMRI) and intracortical recording has yielded important information about metabolic mechanisms that are most highly correlated with recorded neural activity. Microelectric neurochip developments seek to combine signal sensing and processing for bidirectional BMI systems. Studies in the feasibility of using adaptive input-output models for reconstructing hand trajectories have recently been published. These studies have focused on the 18 Chronic devices are intended for long-duration implant, in contrast with acute implanted devices, which are utilized for a short duration and then removed . Formation of scar tissue is dependent on the material used for the electrodes. UNCLASSIFIED//FOA OFFI&IAk YSIE 8HL'I 14 UNCLASSIFIED/,SfOR 8FFI€il.t.k Wlili QPIIL¥ microelectric design of chronic neural implants. In 2006, Fetz et al. illustrated an intracranial operated neural implant that could function as neural prosthetic to assist upper limb movement for individuals that suffered from spinal cord injuries. The intriguing part of the Fetz design is its ability to function as an autonomous battery operated device. This is a departure from traditional BMI devices that are cumbersome and difficult to operate. Clearly this is a step forward in the development of microwire electrode design applications (Reference 44). Figure 5 illustrates the schematic flow chart of the Fetz MEMS neurochip, also called a programmable system-on-chip (PSoC) architecture. Programmable System on Chip Architecture (1R module) ( eITM>ry) ( MeITM>ry) t t l Prim ry ( Secondary PSoC PSoC ( ) J l ! l o :5"' E ~w ;;:, r.n lf n M icrowire e lectrodes EMG ecllod s Figure 5. Schematic of the Neurochip Functional Blocks. Parallel PSoC microcontrollers record neural and muscle signals to independent memory modules. Primary PSoC also controls a constant-current stimulator circuit and communicates via IR to a PC or hand-held PDA. (Reference 44) Traditional PSoC designs operate on a high power consumption and short battery life of radiotelemetry systems (Reference 46). In the Fetz study, researchers implanted electronic devices that collected data in unrestrained monkeys. The system operated without the power limiting and cumbersome restraints exhibited by these previous systems. For this design, an onboard spike processing and a stimulator circuit were employed to allow for real-time bidirectional interface with the nervous system. After several modifications, Fetz et al. successfully created a device that was able to simultaneously record electromyogram (EMG) activity and neural activity at the implant. The lithium battery operated power source and electronic data systems were skillfully packaged within a compact percutaneous (inner skin) titanium casing that was attached to a monkey's skull; the entire implant weighed only 56 g. Neural data was acquired from 12 microwire electrodes chronically implanted in the primary motor cortex. Leads run subcutaneously from the head casing to a connector on the monkey's back. Two pairs of stainless-steel wires were inserted percutaneously into the forearm muscles where they were attached to an EMG signal recorder (Reference 44). UNCLASSIFIED/lf81l OfPltlJlcL U:!I! f>flt I 15 UNCLASSIFIED/ /FOR: orr1e1J1tt Y!H: e ..tv CHRONIC NEURAL IMPLANTS AND FMRI While developing microelectric neural chips for functional studies appears revolutionary on the surface, it only presents a piece to the greater puzzle of understanding the feasibility between neuroprosthetic chips and BMI development. For a BMI system to succeed, researchers must understand the brain, its regions of activity and how those area correlate to real time stimulation and neural responses, and how the brain may evolve with training on use of the BMI. This will require the employment of sensitive neural mapping devices such as fMRI. ~ II { I ! Ill t , i: 0 ..J 04-1.eft Figure 6. Implant Location. Cartoon shows location and orientation of the different electrode sites in the various layers of an animal's neocortical implant (layer thicknesses are approximately scaled). The gray band is the 200µm separation region between the upper and lower layers of the neocortex. To address this question, studies performed by Parikh et al., have targeted tissue­ specific layers within the motor cortex region, essentially a modified ECoG, where penetrating microelectrodes exhibit the greatest functionality in cortical prosthetic design (Reference 47). The interesting parameter of this experimental design is its ability to successfully employ a behavioral task paradigm where electrodes could accurately record brain activity and animal response based on audible and visual cues. With this information the researchers discovered that the lower layers of the cortex are more likely to encode dire