BRAIN-BODY SENSORY FUSION: MERGING FNIRS/EEG NEUROIMAGING AND FULL-BODY MOTION CAPTURE

Functional connectivity between the brain and body kinematics has not been largely investigated due to the requirement of being motionless in neuroimaging techniques such as functional magnetic resonance imaging (fMRI). The importance of investigating this connectivity arises from the fact that the connectivity is disrupted in many neurodegenerative disorders such as Parkinson’s Disease (PD). PD is a neurological progressive disorder characterized by movement symptoms including slowness of movement, stiffness, tremors at rest, and walking and standing instability. Body kinematics are generally divided into two categories of fine and gross motor tasks. Fine motor movements are referred to small range kinematics of the body such as finger movements. Gross motor movements are related to bigger range movements such as limb motion. Unified Parkinson’s Disease Rating Scale (UPDRS) is a protocol consisting of 42 tasks that the patients need to perform for the neurologists to make a decision regarding the diagnosis or progression of the disease. 14 of these tasks are motor exams including both fine and gross movements. In this work, a smart wearable glove has been developed in order to measure fine motor movements of fingers. The performance of the glove regarding measuring finger movements has been tested and reported. The smart glove is tested on a robotic hand and healthy humans performing finger tapping task. The variability in human subjects compared to the robotic hand is clear based on the results of the test. The variance of finger tapping frequency on robotic hand is less than 6.00× e−5 compared to the variance for human participants which is in the range of 0.001−0.1. A non-optical motion capture system (Mocap) consisting of inertia measurement unit (IMU) sensors has been utilized to record the gross movements of the body. Developing a framework for appropriately collecting simultaneous data from functional near-infrared spectroscopy (fNIRS) neuroimaging system and motion capture system is discussed as the first aim. An application protocol interface (API) is developed in order to call each system’s software and record the data on separate files in milliseconds accuracy. The performance of this framework is tested at the system level to validate the data and also tested in real life experiment. A synchrony test is performed in order to validate if the recorded data from these systems through the API is synchronized. The results show that the data is acceptably synchronized with a time difference of few milliseconds which is negligible in fNIRS studies. The real life experiment of performing some gross motor tasks by healthy participants revealed that the brain activity and body movements detected through these systems are coupled. The promising results suggest that this framework can be developed for more complicated processing and experiments. The second aim of this study is to develop algorithms in order to fuse and process the data recorded from the brain and body through the developed framework, and also developing a visualization user interface to monitor the brain activity and body kinematics simultaneously. Algorithms are developed in order to synchronize the recorded data, compensate the uneven sampling rates, calculate oxygenated hemoglobin levels in the brain from fNIRS, calculate the acceleration vector from Mocap, and normalize the flex sensor data from the smart glove. Validating the developed interface and algorithms in a human study is followed as the third aim of this study. 11 PD patients and 10 neurotypical (NT) healthy older participants were recruited in order to perform 4 of the UPDRS motor tasks including both fine and gross movements. The tasks include finger tapping, hand flipping, arm movement, and foot stomping on both left and right side limbs. Each task was explained for the participants and the participants were asked to practice each task. There was no recording during the practice time. Montreal cognitive assessment test and a questionnaire regarding the history of the disease is given to PD participants for further analysis by the neurologists. Body kinematics were recorded by the Mocap and smart glove. fNIRS and electroencephalogram (EEG) have been utilized as the brain neuroimaging systems. The hypothesis was: "There are differences between older healthy adults and PD patients which could be detected and distinguished by the fusion of brain activation and body movements" The recorded data from each of the modalities have been analyzed individually, and the processed data has been used for classification between PD and NT group. The average changes of oxygenated hemoglobin from fNIRS, power spectral density of EEG in the Theta, Alpha, and Beta bands, acceleration vector from Mocap, and normalized flex sensor data were used for classification. 12 different support vector machine (SVM) classifiers have been used on different datasets such as only fNIRS data, only EEG data, hybrid fNIRS/EEG data, and all the fused data for two scenarios of classifying PD and NT based on each activity individually, and all the fused data together. The PD and NT group could be distinguished by the accuracy of more than 85% for each individual activity. For all the fused data, the accuracy of classification for differentiating PD and NT groups are 81.23%, 92.79%, 92.27%, and 93.40% for the fNIRS only, EEG only, hybrid fNIRS/EEG, and all the fused data, respectively. The results show that adding the data from each modality in the classification increases the accuracy, which implies that each modality carries useful information that results in a higher rate of distinguishing PD and NT. The lower accuracy of the fNIRS data only classification is due to the delay of the hemodynamic response which is variable for each individual subject. This makes it hard for the classifier to distinguish the two groups. The promising results show the feasibility of using this brain-body fusion system to distinguish the differences in PD and NT group regarding the fused information of brain and body. The future goal is to develop an interface to provide the visualization of brain activity and body kinematics of the participant simultaneously while the participant is performing the motor tasks. This is happening by a visual observance of the patients by neurologists. The visualization interface can provide a tool for neurologists to observe both body movements and brain activity simultaneously. Also, adding the results of the analysis will be provided at the end of the experiment which can be used as a supporting tool for neurologists to make a decision about diagnosis or progression of PD. Considering more individual information from the patients such as their UPDRS scores, type of medication, or dosage of medication in the machine learning analysis can result in a multi-class classification capable of distinguishing each patient individually. This approach can be helpful in monitoring the progress of the disease or considering the effectiveness of the medication. This needs a quantitative evaluation of sample size calculation and longitudinal data collection in order to generalize the results of this dissertation.

as finger movements. Gross motor movements are related to bigger range movements such as limb motion. Unified Parkinson's Disease Rating Scale (UPDRS) is a protocol consisting of 42 tasks that the patients need to perform for the neurologists to make a decision regarding the diagnosis or progression of the disease. 14 of these tasks are motor exams including both fine and gross movements. In this work, a smart wearable glove has been developed in order to measure fine motor movements of fingers. The performance of the glove regarding measuring finger movements has been tested and reported. The smart glove is tested on a robotic hand and healthy humans performing finger tapping task. The variability in human subjects compared to the robotic hand is clear based on the results of the test. The variance of finger tapping frequency on robotic hand is less than 6.00 × e −5 compared to the variance for human participants which is in the range of 0.001 − 0.1. A non-optical motion capture system (Mocap) consisting of inertia measurement unit (IMU) sensors has been utilized to record the gross movements of the body.
Developing a framework for appropriately collecting simultaneous data from functional near-infrared spectroscopy (fNIRS) neuroimaging system and motion capture sys-tem is discussed as the first aim. An application protocol interface (API) is developed in order to call each system's software and record the data on separate files in milliseconds accuracy. The performance of this framework is tested at the system level to validate the data and also tested in real life experiment. A synchrony test is performed in order to validate if the recorded data from these systems through the API is synchronized. The results show that the data is acceptably synchronized with a time difference of few milliseconds which is negligible in fNIRS studies. The real life experiment of performing some gross motor tasks by healthy participants revealed that the brain activity and body movements detected through these systems are coupled. The promising results suggest that this framework can be developed for more complicated processing and experiments.
The second aim of this study is to develop algorithms in order to fuse and process the data recorded from the brain and body through the developed framework, and also developing a visualization user interface to monitor the brain activity and body kinematics simultaneously. Algorithms are developed in order to synchronize the recorded data, compensate the uneven sampling rates, calculate oxygenated hemoglobin levels in the brain from fNIRS, calculate the acceleration vector from Mocap, and normalize the flex sensor data from the smart glove.
Validating the developed interface and algorithms in a human study is followed as the third aim of this study. 11 PD patients and 10 neurotypical (NT) healthy older participants were recruited in order to perform 4 of the UPDRS motor tasks including both fine and gross movements. The tasks include finger tapping, hand flipping, arm movement, and foot stomping on both left and right side limbs. Each task was explained for the participants and the participants were asked to practice each task. There was no recording during the practice time. Montreal cognitive assessment test and a questionnaire regarding the history of the disease is given to PD participants for further analysis by the neurologists. Body kinematics were recorded by the Mocap and smart glove. fNIRS and electroencephalogram (EEG) have been utilized as the brain neuroimaging systems. The hypothesis was: "There are differences between older healthy adults and PD patients which could be detected and distinguished by the fusion of brain activation and body movements" The recorded data from each of the modalities have been analyzed individually, and the processed data has been used for classification between PD and NT group. The average changes of oxygenated hemoglobin from fNIRS, power spectral density of EEG in the Theta, Alpha, and Beta bands, acceleration vector from Mocap, and normalized flex sensor data were used for classification. 12 different support vector machine (SVM) classifiers have been used on different datasets such as only fNIRS data, only EEG data, hybrid fNIRS/EEG data, and all the fused data for two scenarios of classifying PD and NT based on each activity individually, and all the fused data together. The PD and NT group could be distinguished by the accuracy of more than 85% for each individual activity. For all the fused data, the accuracy of classification for differentiating PD and NT groups are 81.23%, 92.79%, 92.27%, and 93.40% for the fNIRS only, EEG only, hybrid fNIRS/EEG, and all the fused data, respectively. The results show that adding the data from each modality in the classification increases the accuracy, which implies that each modality carries useful information that results in a higher rate of distinguishing PD and NT. The lower accuracy of the fNIRS data only classification is due to the delay of the hemodynamic response which is variable for each individual subject. This makes it hard for the classifier to distinguish the two groups.
The promising results show the feasibility of using this brain-body fusion system to distinguish the differences in PD and NT group regarding the fused information of brain and body. The future goal is to develop an interface to provide the visualization of brain activity and body kinematics of the participant simultaneously while the participant is performing the motor tasks. This is happening by a visual observance of the patients by neurologists. The visualization interface can provide a tool for neurologists to observe both body movements and brain activity simultaneously. Also, adding the results of the analysis will be provided at the end of the experiment which can be used as a supporting tool for neurologists to make a decision about diagnosis or progression of PD. Considering more individual information from the patients such as their UPDRS scores, type of medication, or dosage of medication in the machine learning analysis can result in a multi-class classification capable of distinguishing each patient individually.  The protocol of the PD vs. NT experiment. 10 seconds of activity followed by 10 [1,2,3,4,5], which in turn affects sensorimotor control and functions, then affecting daily life activities [6]. Studies have shown that the dopaminergic system is affected by aging. In particular, dopamine transmission levels decrease with age [7,8,9] which also happens in Parkinson's disease (PD) [10]. PD is one of the most prevalent hypo-kinetic movement disorders and the second most common neurodegenerative disorder [11]. PD affects patients' motor functions, with symptoms including bradykinesia (slowness of movement), rigidity (stiffness), tremor at rest and action, and walking instability. Therefore, there is a need to examine brain functionality and brain-body coupling in aging adults and PD patients, both for information about healthy aging and early stages of PD. Human brain structures have been explored using brain scanning technologies for decades [12,13,14,15,16,17,18,19]. However, the participants need to stay stationary during scanning with traditional brain imaging methods such as functional magnetic resonance imaging (fMRI) in order to avoid motion artifacts. In MRI, these usually appear as ghosting artifacts which obscure useful clinical information [20]. Therefore, there is a lack of knowledge about human brain processes which are tightly coupled to body movements. In order to better understand the functional connectivity between the brain and the body, this doctoral research aims to establish and validate an infrastructure for fusing neuroimaging and body motion capture data.

Research Innovation
Neuroimaging and body motion capture techniques have been used individually for people with movement disorders [21,22,23,24,25,26,27,28,29]. While this allows for an understanding of neurological activities and movement issues, but functional connectivity between the two is not yet understood.
Some studies have investigated the coupling between the brain and the body, but the focus of these studies has been on a few motor tasks related to finger flexion extension or muscle contractions and small movements. Electroencephalography (EEG) or magnetoencephalography (MEG) are the most common brain monitoring systems been used in these combined studies [30,31,32,33,34,35,36,37]. These systems are very sensitive to movement artifacts, preventing their use in experiments involving bigger range movements.
To develop a system which allows larger movements to be combined with neuroimaging, a brain monitoring subsystem needs to be chosen that would allow the participants to move during the experiment without making motion artifacts. In order to meet this criterion, functional near-infrared spectroscopy (fNIRS), and EEG have been chosen to be used as the brain monitoring systems. The fNIRS system captures hemodynamic responses in the brain and has been shown to be less prone to motion artifacts.
The drawbacks of this system are low temporal resolution and the delay in hemodynamic response. In order to compensate for these drawbacks, EEG is added to this research which provides a higher temporal resolution (256Hz compared to 10Hz) and faster electrical activity responses.
Body kinematics are generally divided into gross motor movements and fine motor movements. A non-optical motion capture system (Mocap) is used to measure and record gross movements of both upper and lower limbs. The Mocap system captures full-body movements, but it is limited to gross movements, such as limb movements like hand pronation/supination, arm movements, and foot stomping. This system is not able to measure fine motor movements such as finger tapping, one of the most common tests in PD screening. A smart glove integrated with flexible sensors has been developed in order to measure fine motor finger movements. Hence, adding it to this study can provide more information about the coupling of brain and body for fine motor movements. Figure 1 shows an overview of this research proposed fusion system. The initial work in this doctoral dissertation included the development of a smart glove capable of measuring fine finger kinematics. The development of the infrastructure for fusing neuroimaging and body motion capture followed. It was then possible to create simultaneous processing algorithms for brain activity and body movements.
Finally, a human study on PD patients and healthy participants was done to evaluate the fusion system and associated algorithms with the goal of distinguishing these two groups based on the recorded fused data.
Overall, the activities of this doctoral dissertation work can be divided into the following three aims: Aim 1: Development of a multi-modal brain-body fusion system The first aim of this dissertation is to develop infrastructure for the fusion of multimodal brain-body recordings. In general, this system consists of two main components: brain monitoring and body motion capture. fNIRS and EEG will be used as neuroimaging modalities. A Mocap system and the smart glove will be used in order to record gross movements and fine motor tasks, respectively.
Each of these systems has their own native data recording software. The EEG data acquisition system has a trigger input which allows for connection and synchronization with the other systems, and therefore does not require further development for fusion.
Fusion system development therefore focuses on the fNIRS and Mocap systems which allows the other systems to be connected and synchronized with. An application protocol interface is developed in order to connect to each of these systems and record simultaneous data from these systems.

Brain-Body Synchrony
The second aim is to develop algorithms and models to quantify the synchrony between the brain and the body based on multi-modal data recorded from the fusion system developed in Aim 1. Algorithm development can be divided into two tasks as follows: 1. Developing signal processing methods in order to process brain signals from fNIRS and EEG and body movements from the Mocap and the smart glove. This includes algorithms to calculate the level of oxygenated hemoglobin from fNIRS data, the power spectral density of the EEG data, the acceleration vector of the Mocap, and the normalized voltage of the smart glove.
2. Developing data visualization algorithms in order to visualize neural activation and body movements together.
Aim 3: Human study to evaluate the brain-body fusion system and associated algorithms This final step of the study is needed to validate the performance of the first two aims and investigate the hypothesis that "there are differences between older healthy adults and PD patients which can be detected and distinguished by the fusion of brain signals and body movements." In this aim, two experimental groups will be tested for evaluation and validation of the developed systems. The two groups of participants are 1) 11 individuals with PD and 2) 10 neurotypical (NT) healthy older adults. Body kinematics were recorded by Mocap system and smart glove, and brain activity was recorded by both fNIRS and EEG. The developed algorithms from Aim2 were used to measure relevant activity from each individual system, and the two groups of participants were distinguished using support vector machine (SVM) classification method.
This research will have a profound impact on neuroscientists' ability to understand brain-body coupling for individuals with neurological disorders. The research activities will create a framework for simultaneous brain-body sensing, which will provide new tools and algorithms to the scientific community. In addition, the project will provide foundational knowledge for future health care devices, new methods of treatment, and more accurate diagnoses.  Longitudinal magnetic resonance imaging studies of older adults: a shrinking brain," Journal of Neuroscience, vol. 23, no. 8, pp. 3295-3301, 2003.

CHAPTER 2
Background This chapter provides more detailed information about the target population of this work, people with PD. Since this dissertation involves measuring body kinematics for fine and gross motor movements and utilizing brain neuroimaging systems, fNIRS and EEG, each of these procedures is explained more in detail in the following sections.
Each section highlights selected related works.

Parkinson's Disease (PD)
Parkinson's Disease (PD) is the second most common neurodegenerative motor disorder, affecting 4 million [1] people worldwide. Studies suggest that approximately 0.6% of the total United States population, and 0.8% of the European and Canadian population will be affected by PD by 2050 [2]. PD patients have various movement-related symptoms including tremor at rest, muscle stiffness or rigidity, slowness of movement, and gait and balance problems such as shuffling, freezing or falling because of alterations in their brain-body coupling.
PD is commonly diagnosed between age 50 and 70. The pathological hallmark of PD is the progressive degeneration of dopamine-secreting neurons in the midbrain structure known as the basal ganglia [3]. The basal ganglia, like other brain areas, manifests electrical activity that becomes synchronized with connected structures. The basal ganglia regulates voluntary movements through connections to the thalamus and the motor cortex. The degeneration of dopamine neurons in the basal ganglia affects neuron firings that transfer commands from the brain to the rest of the body. This effect on the motor pathways results in movement symptoms such as tremors, rigidity, stiffness, dyskinesia or slowness of movement, and imbalance in body posture as can be observed in PD. Figure 2 shows the efferent pathway of neural activity from the brain to the limbs.  [4,5].
Making an accurate diagnosis of PD, especially during early stages, is difficult since there is no standard diagnostic test for Parkinson's. In general, physicians conduct screening tests to visually observe the presence or absence of relevant symptoms [6]. The Unified Parkinson's Disease Rating Scale (UPDRS) [7] is one of the most commonly used protocols for PD diagnosis and monitoring disease progression. The UPDRS protocol includes 42 questions including 14 movement exercises that allow physicians to determine if someone has PD or not. Functional neuroimaging techniques such as fMRI are not commonly used in diagnostic screening due to the movement tasks of the UPDRS and requirement for stillness in fMRI scanning. However, these neuroimaging techniques are heavily used in research on the neurological processes and predictors of PD [8,9].
PD has been associated with alterations in brain-body coupling, also known as cortico-muscular coherence (CMC) [10]. Affected neuromuscular pathways result in problems in transferring signals from the body to the brain and vice versa, affecting brain-body coupling and measurable with CMC. Coherence is the degree of time-locked correlation between two signals as a function of frequency [11]. CMC usually refers to coherence between cortical signals and electromyographic (EMG) muscle contraction measurements. Movement sensors such as accelerometers can also be used to measure coherence between cortical signals and body movements, as in cortico-kinematic coherence (CKC). Some abnormal patterns of the CMC has been reported in movement disorders such as PD [12,13].

Brain-Body Synchrony Related Works
Most studies of CMC have utilized EEG or magnetoencephalography (MEG) as the brain monitoring technique and EMG in order to record the muscle contractions [14,15,16,17,18,19].
Recently, Sridharan et al. [20] measured the CMC between the MEG and EMG from six PD patients undergoing deep brain stimulation (DBS) ON and six medicated PD patients (levodopa) along with ten age-matched healthy controls. The protocol of the experiment was hand gripping. Sridharan et al. [20]  Caviness et al. [11] observed an abnormal increase in the CMC of the PD patients with small amplitude cortical myoclonus. The study involved PD patients with and without myoclonus and controls. Coherence peaks were observed in the 12 − 30Hz band in PD patients with myoclonus which were significantly greater than the PD patients without myoclonus and controls. Caviness et al. [11] findings provide evidence that there are abnormal rhythmic activities in cortical motor areas in PD patients with myoclonus.
Marty et al. [22] measured the CKC between MEG and a 3 − axis accelerometer, while eleven healthy adults executed or observed a goal-directed hand action performed by an actor in front of them. Statistically significant coherence at the movement frequency and its first harmonic was observed in both conditions. This significant coherence is reported in the visual cortices, right posterior superior temporal gyrus, bilateral superior parietal lobule, and primary sensorimotor cortex.
Bourguignon et al. [23] presented CKC as a promising method for reliable and convenient functional mapping of the human motor cortex. Coherence between MEG and a 3 − axis accelerometer on the right index finger showed peaks around 3 − 5Hz and 6 − 10Hz, corresponding to the self-paced flexion-extension of the finger at 3Hz. Coherence was significant for all ten subjects in the contralateral hand area of the primary motor cortex.
In another study by Bourguignon et al. [24], CKC between MEG and a 3 − axis accelerometer on the right index finger during fast repetitive voluntary hand movements was measured for ten healthy right-handed adults performing the flexion-extension movement of right-hand finger with and without thumb-finger touching. It was reported that the coherence values were significantly higher in the touch experiment compared to no touch, with the main sources of coherent activity in the left primary motor and sensory hand areas.
Another report by Bourguignon et al. [25] investigated CKC between MEG recorded from a participant viewing experimenter motion and the 3 − axis accelerometer on the index finger of the experimenter. It was demonstrated that a significant peak in the coherence spectra at the flexion-extension frequency of 3Hz and its first harmonic is observable strongly in the visual areas, and also in the primary motor cortices of both hemispheres and in the cerebellum.
All these studies examine the coupling between the brain and the body and how the correlation and coherence between brain activity and body motion are changed in PD and healthy groups. These studies do not differentiate between the two groups. This dissertation study focuses on developing a multi-modal brain-body fusion system and distinguishing PD participants from healthy participants.

Full Body Motion Capture
Motion capture, a process of recording movements, is used in a variety of applications including medical, sports, and entertainment [26,27]. Body movements can be divided into gross and fine motor movements. Gross movements refer to bigger range kinematics such as limb movements or walking. Hand flipping, arm movements, and foot stomping are gross motor tasks. Fine movements refer to small movements such as finger motion. Finger tapping and fist opening-closing are examples of fine motor tasks.

Fine Motor Movements
Small movements of the body such as finger movements are important to monitor due to the fact that people with movement disorders such as PD have difficulties in performing these movements. Motion symptoms of PD affect the performance of some movement tasks involving fine motions. As an example, tremor and rigidity in PD affect the performance of smooth finger tapping, which is shown in this study. It also affects daily life activities such as writing or grasping an object. Finger tapping is a standard and well-established method for assessing the state of disease progression or treatment in patients with PD. The movement requires fine motor control, the ability to keep track of time and tempo, and the ability to clasp and tap [28,29,30,31,32,33].
Measuring these fine motor movements requires special devices capable of recording small movements precisely. Wearable devices integrated with sensors could be a good tool for this purpose. Given their potential for improving health care, wearable devices have been proposed and used in the treatment of PD. As listed in Table 1, there is a large body of research works dedicated to wearable technology and its applications in PD or other movement disorders. The research works can be divided into two categories: 1. Wearable systems design that aims to make the sensing approach more specific to motor symptoms, 2. Algorithms that use off-the-shelf wearable systems to improve the detection, quantification, and classification of the symptoms.
A growing number of commercial products are emerging in the market to fill the gap between the clinical practice and telemedicine.  [50] and conductive paints [51]. Smart textiles refer to a broad range of research activities to integrate advanced functionalities such as sensing, actuation, computing, energy harvesting, communication, and human-computer interaction into textiles [52,53,54,55,56,57]. Based on the functionality, smart textiles can be classified into two subgroups: 1. Passive e-textiles, which have only sensing capabilities for users or environmental conditions. For example, passive (battery-less) UHF RFID (radio frequency ID) tags are woven into regular clothing for indoor human motion tracking [58] and fetal monitoring [59].  [34,35,36] medication or deep brain stimulation (DBS) in patients with PD or essential tremors. Researchers achieved a high accuracy for predicting tremors in PD patients. A sensor node called "Throttle Gyro" Mercury [37] was developed to provide automated activity detection at low-energy consumption. eGaIT is an automated gait (walking) eGait-Smart Shoes analysis system, consisting of [38,39] a shoes pair with embedded inertia sensors. The shoes pair helps analyze and classify abnormal strides during UPDRS gait task. A number of smart gloves have been Various other Smart Gloves proposed and tested to monitor dyskinesia [40,41,42,43] or tremors in PD. Some of the projects have adopted flex sensors to measure tremors on individual fingers. Previously, our research group developed a smartwatch-driven solution to quantify SPARK the motor symptoms of PD. We studied this a smartwatch-based system approach on 24 patients with PD in a [44] clinical focus group study. Although it produced a higher accuracy, it was not reliable for fine-motor symptoms which are generally observed in the finger tapping task.

Products
Kinesia One is a product consisting of a ring sensor that streams the finger motion data to a smartphone.

Kinesia Products [45]
Kinesia 360 is a multi-sensor system for patients to wear at home. Physicians have access to the tremor information for making informed decision about medication. The logger looks like a wristwatch PKG Data Logger [46] and collects movement data remotely from patients with movement disorders to assist doctors with the diagnosis and treatment.
2. Active smart textiles, which have both sensing and actuation capabilities with a power source on board.

Gross Motor Movements
Gross motor movements refer to larger movements such as limb movement or walking. People with movement disorders such as PD have difficulties performing gross motor tasks. Rigidity, bradykinesia, gait and balance problems affect the usual smooth performance of gross motor tasks. Gross motor tasks are generally recorded by motion capture systems.
Generally, motion capture systems fit into two different categories of Optical Motion Capture (OMC) or Non-optical Motion Capture. In optical motion capture systems, markers are placed in the desired locations where the movement should be recorded, which are the joints in the medical human movement studies. Several infrared cameras then collect the movements of the markers and by data fusion techniques, human movement can be observed [60,61]. Although optical motion capture systems provide high accuracy, they have some disadvantages, such as cost and environment dependency [62,63].
Non-optical motion capture systems do not use cameras to record movements. A widely used non-optical system is the inertial system which uses inertia measurement units (IMUs) to record movement. IMUs utilize a gyroscope, magnetometer, and accelerometer in order to record movement and send data to a computer wirelessly. Figure   3 shows an IMU sensor with three degrees of freedom. In addition to having a lower cost than OMC systems, IMU systems do not require cameras, allowing them to be used outside infrared-equipped labs [63].

Neuroimaging
Non-invasive studies of brain functions are more challenging to conduct in humans because several current neuroimaging modalities such as fMRI, single-photon emission computed tomography (SPECT) and positron emission tomography (PET) are costly, bulky, and immobile. fMRI, one of the most widely used neuroimaging techniques, observes blood oxygen level dependent (BOLD) signals in the brain and relies on the observation that the hemodynamic response and neuronal activity are coupled [64]. fMRI confines patients or study participants into a narrow tunnel where individuals remain motionless, except possibly for small maneuvers such as finger tapping or ankle rotations. The temporal resolution in fMRI is poor, producing one image per second, while the brain's electrical and hemodynamic activities happen at a much faster rate. Figure 4 shows a graphical comparison of the widely used neuroimaging techniques. As investigations of coupling between the brain and body movements require participants to be mobile, systems like fMRI are not suited to this purpose. Two portable neuroimaging techniques that can provide mobility for the participants, fNIRS and EEG, have been used in this study. Utilizing these two systems together provides useful information about brain activity during movement and incorporates both the good temporal resolution of EEG and the optimal spatial resolution of fNIRS.

Functional Near-Infrared Spectroscopy (fNIRS)
The feasibility of using near-infrared (NIR) light to illuminate brain tissues has been demonstrated by experiments on cat heads [65]. These experiments showed the capability of NIRS to monitor changes in hemoglobin oxygenation of blood. This is the main chromophore (the light absorber components of tissue) affecting the NIR light absorption and scattering [65]. Therefore, NIR light is employed in fNIRS systems because 1. Most human body tissues are almost transparent to NIR light [65,66], 2. Existence of oxygen in the tissue results in absorption of light [67].
In fNIRS, light emitting diodes (LED) emit NIR light that interacts with the chromophores on a banana-shaped path (like a parabolic path) through the cortex, and a photodetector receives the attenuated light that reaches the skin [68]. Figure 5 shows the path of the light through the cortex. Light propagation, in general, depends on the optical properties governing reflection, scattering, and absorption. The absorption and scattering are influenced by wavelength. Reflection, on the other hand, is related to the angle of the tissue and the light [65].
Light scattering is higher for low wavelengths, so NIR light is scattered less in the body than visible light is. The light within 700nm − 1300nm wavelength range has the best transmission through scalp and skull tissue due to high absorbance of light by hemoglobin. Scattering increases in wavelengths below 700nm and water absorbs most light in wavelengths above 1300nm [65].
The energy of the absorbed light is wasted as thermal energy through the absorber material whose molecular properties determines the specific wavelength at which absorption occurs [69]. The most important chromophores for fNIRS are oxygenated hemoglobin (HbO2), deoxygenated hemoglobin (Hb) and cytochrome c oxidase, since their concentration changes with time and oxygen level [67]. The extinction coefficient (ε) is a function of the wavelength and shows to what extent the chromophore absorbs light, and results in different absorption spectrum for each chromophore [70].
Cytochrome c oxidase varies with oxygen level, but its concentration does not completely depend on changes in oxygen level. It is a mitochondrial enzyme which describes intracellular oxygenation [71]. Therefore, Cytochrome c oxidase is not used as a chromophore to monitor the oxygenation of the tissue. For fNIRS, the interest is in Hb and HbO2 which are responsible for oxygen delivery and carbon dioxide removal in the body. From Figure 6, the reason for choosing the wavelength of the light in the range of 650 to 950nm is shown. In this range, Hb and HbO2 have unique spectra, and water absorption and scattering remain low. Above 950nm the absorption of water increases, resulting in more scattering. Below 650nm, Hb has a higher absorption rate. Therefore, in this range, water absorption is low, and almost all the absorption is on Hb and HbO2, as desired. Figure 6 shows the absorption coefficients of Hb and HbO2. Figure 6: Absorption coefficient of Hb, HbO2 and water adapted from [72].
The attenuation of light can be caused by scattering as well as absorption. In the scattering of the NIR light, photons change direction without energy loss. The change of the direction depends on different aspects, such as wavelength, the size of the particle and the refractive indices of the layers which the photon is going through. The structure of the human head is complex, as it is composed of many layers with varying densities and thicknesses. These layers scatter NIR light in different ways. The vast majority of the solid part of the brain tissue is made of red blood cells which is around 1.5% of the tissue, therefore they are the major source of scattering. Most scattering in the head is due to skin, bone, or cerebral white matter. [67].
Both absorption and scattering should be considered when using NIRS. The Beer-Lambert law relates the attenuation of the light to the chromophores concentration: A is the attenuation of the light which is the logarithmic ratio of the intensity of the received light (I) to the intensity of the source light (I o ). ε is the molar extinction coefficient, C is the chromophore concentration and d is the distance between optodes. The Beer-Lambert law can be expanded to include multiple particles, such as chromophores in NIRS, as follows: In NIRS studies, it is assumed that d and ε 1 , ε 2 , ..., ε n are known. The Beer-Lambert law is then used to find C 1 ,C 2 , ...,C n by measuring A. At least as many wavelengths are required as chromophores in order to accurately calculate the concentration of the chromophores. These wavelengths should be chosen in order to improve the sensitivity of the estimate, such as by choosing one wavelength to maximize the molar extinction coefficient of each chromophore, Hb and HbO2. Therefore, the wavelengths can be found on either side of 800nm, where Hb and HbO2 have equal molar extinction coefficients (see Figure 6). This is called the isosbestic point. It is also possible to increase the number of wavelengths to measure the concentration of other chromophores such as cytochrome c oxidase and water, or to improve the accuracy of Hb and HbO2 concentration measurements [71,73].
It is important to notice that if d is not equal to the distance between the source and detector, it is not possible to quantify the changes in the hemoglobin, and scattering will change it with no pattern. Assuming this fact, the light travels a greater distance than d which is called differential path length (DP). DP is the actual distance that light travels from the source, passing through the tissue and reach the detector. To modify the Beer-Lambert law, the differential path-length factor (DPF) is introduced [74] as follow: Therefore, the modified Beer Lambert law, using DPF is as follow: In the modified Beer-Lambert law, the attenuation is not linearly related to the extinction coefficient because of the unknown term G which includes the effects of the shape of the optodes and the scattering factor. Therefore, it is not possible to calculate the exact concentration of the chromophores using the modified Beer-Lambert law. By using mathematics including differential equations and the assumption that G is constant across all chromophores, it is possible to eliminate G from the equation and calculate changes in the chromophores concentration [73]. The changes in the concentration of the chromophores can then be found with the assumption that d and DPF are constant over the experiment.
It is feasible to determine NIR wavelengths that can minimize the error in calculations carried out by this equation, introduced by the assumptions above [75]. The NIR photons propagation through tissues and therefore optical path length is essential information to accurately calculate the HbO2 and Hb concentrations. It is usually estimated with experimental techniques such as time domain (TD), frequency domain (FD) [76] methods or mathematical modeling such as Monte Carlo (MC) simulations [77] and finite element analysis [78]. As fNIRS studies are mostly interested in two chromophores, Hb and HbO2, by using two different wavelengths and measuring the changes in the attenuation of the light, it is possible to measure the changes in the concentration of the chromophores.

Electroencephalography (EEG)
EEG is a non-invasive electrophysiological monitoring method measuring electrical activities from the brain. The electrical activity is the result of the summation of inhibitory and excitatory postsynaptic potentials at the dendrites of neuron ensembles.
When neurotransmitters activate the ion channels on the cell membrane, ions flow in and out of the neuron resulting in potential changes, which create electrical fields around the neuron. Single neuron electrical fields are too weak to be measured by non-invasive EEG recordings. However, when thousands of neurons have synchronized activities, the electrical fields will sum and generate strong fields. The electrical conductivity of brain tissues, skull, and scalp are different, but there is enough conductivity among these tissues to transmit these generated electrical fields to the outer layer of the scalp, where they can be detected by EEG [79,80,81]. However, air has almost no electrical conductivity. Therefore, the EEG electrodes need to be in direct contact with the skin, or additional conductive material is needed in order to fill the air gap between electrodes and skin. This material can be electro-conductive gel or paste, which is common in EEG scalp recordings.
EEG has been used in many different applications such as epilepsy, sleep disorder diagnosis, brain death, tumors, stroke, and many other disorders [82]. The spatial resolution of EEG is lower than that of other neuroimaging techniques, such as CT, PET, MRI, and fNIRS. However, it's portability and good temporal resolution are significant advantages. The high temporal resolution of EEG provides several frequency ranges in the data. Some common bandwidths in the EEG data are as follows: • Delta (< 4Hz): Delta waves are low-frequency signals with high amplitude. Delta waves can be recorded in frontal regions in adults, and posterior regions in children. It is related to normal sleeping.
• Theta (4 − 7Hz): Theta waves are slow. It is higher in younger children, and appears in awake children, but mostly in drowsy adults. It is not common in awake adults.
• Alpha (8 − 15Hz): Alpha waves can be recorded in the posterior regions of the head, and are related to a relaxed mood, or when the eyes are closed. Alpha waves can also be detected in people in comas.
They are usually symmetrically distributed in both hemispheres of the brain, and are common in active thinking, focusing, and anxiety.
• Gamma (30 − 100Hz): Gamma waves are high-frequency oscillations in the brain and can usually be recorded in the somatosensory cortex. They are related to subjective awareness and short-term memory matching of recognized objects. and different frequency bands in EEG [84] In this dissertation, the systems that have been used are as follow: • The next chapter provides more information about these systems along with the system level tests and evaluations on the developed devices and interfaces. The human study experimental design and data analysis approach are also presented.  [45] "Kinesia one," http://glneurotech.com/kinesia/products/kinesia-one/.

Methods & Development
This chapter explains the development of devices and interfaces along with the system level tests. This is followed by the human study experimental design, data analysis approach and classification methods. Figure 8 shows a more detailed schematic of this research. The first section explains the development of WearUp glove and the sensors and microprocessor used in the system. The system level tests used to validate the performance of the glove for measuring fine motor movements are laid out in detail. This section lies in the sensing layer depicted in Figure 8.
The second section describes the systems used for the brain-body fusion interface.
This section concludes with the developed algorithms to synchronize the brain and body data and the system level tests that were performed to validate interface performance.
This section lies in the Fusion Algorithm block of the data analysis layer illustrated in Figure 8.
The third section provides information about the human study experiment with the aim to distinguish PD group from the healthy neurotypical (NT) group.
The last section explains the data analysis from each individual system and introduces the techniques that have been used to classify the PD and NT groups. As this is a multi-modal recording system, some specific features must be extracted from the data, including brain activity power in certain frequency bands, windowed averages of the oxygenated hemoglobin, frequency of finger tapping, and movement acceleration.
These features can be used to measure correlations between brain activation and body movements and in machine learning algorithms to classify subjects into groups. This section lies in the data analysis layer shown in Figure 8.

WearUp Glove Development
WearUp is an active smart glove developed to record and detect patient's hand movements such as finger tapping. WearUP consists of two flexible sensors woven into a base fabric, an embedded processor with Bluetooth low energy for wireless communication of sensor data, and a power management module. The two flex sensors are woven onto the index finger and thumb. They measure the angular deformation when patients perform the finger tapping task commonly used in the clinic to screen PD patients. The sensors used in the WearUp glove are flexible resistive sensors which change their resistance under bending or angular deformation. In other words, by placing the sensors on the fingers, and bending the fingers, an increase in sensor resistance can be detected.

Flex Sensors
The assessment of hand movements requires the device to sense and compute the data reliably and accurately. Assessing the hand's position and movement requires relative spatial orientation of the fingers and palm. WearUP focuses on the use of single angle flex sensors. The flex sensors are variable resistors which use conductive ink. The resistance of this ink increases with its surface area. The sensors are placed between two flexible, but conductive traces housed in a clear insulative material. The resistance when flat is roughly 20kΩ and when bent to a 90°angle increases to roughly 70kΩ. Figure 9 shows the WearUp glove and flex sensor used in development. Law. One last challenge on this setup was maximizing the voltage range to increase the signal to noise ratio. A standard 10kΩ resistor is used, providing a potential voltage swing of approximately 1v. This range was mapped from 0 to 90 as the flex sensors resistance reaches its maximum at a 90°angle.

Intel Curie-Based, Low-Power Embedded System
The microcontroller unit (MCU) must be small enough to fit on the back of the hand, consume minimal power such that the battery life of the glove extends at least beyond one practice session, and have the processing power to compute the simple mapping of the analog to angular values near real-time speeds. Currently, the MCU being used, Arduino101, meets these requirements. The Arduino101 is a learning and development board which contains an Intel Curie Module. This board is designed to integrate the core's low power consumption and high performance with the Arduino's ease-of-use. The Arduino101 also provides 19 channel 12 − bit ADCs, Bluetooth Low Energy capabilities, and power management circuitry to ensure stability in reference to analog readings.

WearUp System Level Testing
The experiment protocol that was used to validate the performance of the glove to detect finger movements is explained in this section.

WearUp: Experiment Protocol
This section presents the procedure of the preliminary experiment to evaluate the performance of the WearUp glove regarding the detection of fine movements of fingers such as finger tapping.

Robotic Hand Platform for Testing the WearUP Glove
In the first set of the experiments, the performance of the glove with a standard, specific measurement with no user variability was tested. To achieve this goal, an equivalent right-handed WearUP glove was created and was placed on a 3D printed robotized robotic right-hand based on the InMoov design as shown in Figure 11. The robotic hand was set to perform finger tapping tasks at two different speeds. For fast finger tapping, the goal was to finish a cycle of finger tapping in 1.5seconds. For slow finger tapping, the goal was to finish a cycle of finger tapping in 3seconds. Since the robotic hand was set for a specific task with a fixed angle of bending the fingers, it was anticipated that the results of the robotic finger tapping experiment should have little variation in sensor's reading from the glove. Therefore, the goal of this experiment was to validate the performance of the WearUp glove.

A Pilot Study on Human Participants
In the second set of experiments, nine healthy human participants were recruited after receiving IRB approval (#1020863-1). As shown in Figure 12, the participants were asked to perform the finger tapping task that involves tapping the pointer finger to the thumb for 15 times at a relatively fast speed. The finger tapping was followed by resting their hands in the fully-opened position for five seconds. The participants were then asked to perform the finger tapping for 15 times at a slower speed. This experimental protocol is shown in Figure 12. The data was collected from each participant for 3 rounds of the experiment. The participants were instructed to perform the task at their own pace to evaluate the performance of the WearUp glove in detecting variations in finger tapping velocity.

WearUP: Algorithms & Methods
This section explains the algorithms that have been used to validate the performance of WearUp.

Algorithm
This section presents the analysis of algorithms that have been developed to evaluate the performance of the WearUP glove in the preliminary experiment. WearUp Glove was used on both robotic hand and healthy human participants.
The protocol of the experiment included the finger tapping task, which required the participants to bend their index finger to complete the task. Therefore, the goal was to detect peaks in the collected data during finger tapping.
The raw data collected by the sensors contain noise from the environment and micro-movements of the hand which are not desired for further processing. To reduce the noise from the data, a 4th order lowpass Butterworth filter was applied. The Butterworth filter, also called maximally flat magnitude filter, is designed to have a flat frequency response in the passband range. The 4th order Butterworth filter has the roll off of −24dB per octave in the stopband frequencies. Figure 13 shows a sample of the raw data, filtered data and depicts the differences regarding the smoothness of data.

Finger Tapping Detection
Since the resistance of the flex sensors increases due to finger tapping movements, periodic peaks have been observed in the data. A peak detection algorithm that is capable of detecting the peak angles in finger tapping was developed. A threshold was set such that if the resistance of the sensor exceeds this value, a potential peak related to the finger tapping is noted. The resistance of the flex sensor is equal to 20kΩ in the straight (non-bending) position and above 70kΩ at the bending of 90°. Therefore, the threshold has been chosen to be 20% bigger than the straight value (1.2 * 20k) to enhance the sensitivity of peak detection. Due to the small changes in the sensor value which are not related to the finger tapping, a temporal threshold has also been set.
This means that if the detected peaks are very close to each other temporally, they are most likely not two different peaks of interest, and represent an "interruption" which is seen and rated on the clinical exam, UPDRS. Therefore, the temporal threshold has been set to be 0.2 seconds. Setting this threshold is reasonable in the sense that healthy humans cannot volitionally tap the fingers more than 5 times per second. Thus, the developed algorithms avoid missing any peaks related to the actual finger tapping.

Tapping Rate/Frequency
The speed and amplitude of the finger tapping task are important in the assessment of PD patients. This information indicates symptom severity and treatment efficacy. In order to provide this information from the WearUP glove, it is important to find the frequency of finger tapping (time interval between each finger tap). To achieve this, the algorithm was tested for detecting the peaks. By detecting the peaks, and finding the number of time stamps in between two consecutive peaks, the time in seconds that has taken for the subject to tap their thumb can be calculated. Knowing the time between each finger tap, allow to calculate the instantaneous frequency of finger tapping.

Application Protocol Interface (API) Development
In this section, the steps and the process for creating the data fusion interface are explained.

Systems
• Brain Monitoring System: The fNIRS system used in this study is the NIRScout system (NIRx Inc., New York, NY, USA). This system utilizes 8 light sources and 8 light detectors and provides the ability to change the optode montage setup. Therefore, the system can be used in any region of the brain. In this study, the optodes are set up in the area of the motor cortex. Also, the 8 * 8 optode montage can provide up to 20 channels of the fNIRS data since fNIRS channels are pairs of nearby sources and detectors. The fNIRS system uses two different wavelengths in order to measure both HbO2 and Hb. Having 20 channels of fNIRS data with two wavelengths for each channel results in a total of up to 40 data attributes. • Body Motion Capture System: In this study, a non-optical motion capture system called YEI 3-Space Mocap (YEI Technology, Portsmouth, OH, USA) is used.
The system consists of 17 IMU sensors that are able to record the movements of different joints of the body. Each YEI 3-Space sensor uses a triaxial gyroscope, an accelerometer, and compass sensors in conjunction with advanced processing and on-board quaternion-based Kalman filtering algorithms to determine orientation relative to an absolute reference in real-time, resulting in 9 data attributes for each IMU sensor. Therefore, the Mocap system provides a total of 153 data attributes while recording whole body movement.

Implemented API
Although both the Mocap and fNIRS systems already have their own native software, Mocap Studio and NIRStar, respectively, building a custom graphical user interface (GUI) was necessary to provide more control over the synchronization of data collection from both systems. To make this GUI, mainly two Python wrappers have been used. The Mocap sensors' wrapper, named threespace, can be found on GitHub [4]. This Python wrapper is compatible with YEI 3-space motion-capture devices and is open-source. The fNIRS wrapper comes as a software add-on to the NIRStar15 that allows data to be streamed through an API called the Lab Streaming Layer (LSL). The API is available in several different languages including Python. This module is also available on Github [5].
Threespace allows the Bluetooth dongles to connect to the wireless motion capture sensors through USB. This allows for data streaming from the sensors through this API. The main interest is with streaming the raw accelerometer, gyroscope, and magnetometer data, each with 3 degrees of freedom (X,Y , and Z). Given the significant number of sensors (up to 17), collecting the data sequentially would not be optimal; for the following sample of every sensor would only be collected after all other sensors have been collected from first. This is overcome by the API through the option of batch data collection. This batch data collection records data simultaneously from all sensors involved. The data collected is written to a comma-separated value (CSV ) file.
Due to the fact that it might not always be desirable to collect data from all the sensors, each sensor has been linked to a Boolean value that is stored in a Python dictionary. Using this method, data is collected from the sensor only when this value is set to True. This feature will come in handy in the construction of the GUI.
The LSL API for fNIRS enables the corresponding module connects to the NIRStar15 software and detects the available fNIRS channels before finally collect-ing the readings from all those channels. Most of the setup is thus done on the native software and the API simply collects the data and writes to a text file. Figure 17 shows an overview of the APIs in this study. Figure 17: Overview of the Brain-Body Fusion software (BBM API) developed.

Data Synchronization
After motion capture and fNIRS data are collected and exported, the data is imported into MATLAB for analysis. The sampling rates for the motion sensors and fNIRS system vary depending on the number of sensors or channels. With the experimental set-up, the approximate sampling rates for the motion sensors and fNIRS system are 20 − 30Hz and 7.81Hz, respectively. However, the Mocap sends data wirelessly, and so both the increments between samples and the number of samples per second vary over time. To correct the uneven sampling rate, each data set is interpolated in MATLAB using a one-dimensional interpolation function to up-sample to an even sampling rate of 50Hz. It is noteworthy that because of the variable sampling rate of the Mocap, the interpolation is performed based on 1second windows. Therefore, starting from the be-ginning of the data, based on the timestamps that the API provides, which is reading the clock of the recording computer in milliseconds, the first 1second of both Mocap and fNIRS data are taken, interpolated to 50 samples per second, and move to the next 1second of the data to be interpolated. This is repeated for the whole duration of the data recording. Figure 18 shows an example of 2seconds data, divided into two 1second windows with different samples per second and interpolated to an even 50 samples per second.

BBM API System Level Testing
This section explains the experiments that have been done at the system level in order to test the performance of the BBM API and the accuracy of the fusion procedure.
There have been two different tests:

Mocap-fNIRS Synchrony Test
The first objective is to observe the synchrony between the Mocap and fNIRS data while the data is recorded through the developed interface. The first part of the experiment is to use a single Mocap sensor and only one pair of the fNIRS optodes, for a single channel of data. In this experiment, both the Mocap sensor and the pair of fNIRS optodes are placed on the arm of the subject and the subject is asked to tap on the desk every 25 seconds. The goal of this experiment is to observe the synchronization between the Mocap and fNIRS data, as tapping on the desk would make a remarkable noise on the data which would be easy to detect.

Brain-Body Sensor Fusion Test
The second objective of this part is to test the developed interface using all of the

Foot stomping for both the right and left legs
The order of the tasks is the same as the order mentioned above starting with hand flipping and finishing with foot stomping. The process of performing the task was explained for the participants, but there was no practice recording before the experiment.
The protocol of the experiment starts with 20seconds of rest (no movement) followed by 10seconds of activity, repeated for 5 times. Therefore, for each activity, the experiment lasts for 2:30 minutes, and for all the six different activities, the experiment takes a total of 15 minutes. Figure 19 shows the experimental protocol. The participants were asked to follow the provided visual cues and perform the activity at their own pace. After data collection, the same method as explained earlier is used to upsample the fNIRS and Mocap data to correct for uneven time increments in each data set. The Mocap data will be checked to detect the timings of the movement and compare it with the timing of brain activation. Figure 19: The protocol of the Brain-Body Sensor Fusion experiment: 20 seconds of rest followed by 10 seconds of activity (Hand Flips, Arm movements, and Leg movements for the left and right side).

PD vs. NT Experimental Design
This section describes the experiments on PD patients and healthy neurotypical (NT) participants in order to measure the synchrony and correlation between brain activity and body movement. fNIRS neuroimaging and body motion capture have been recorded through the BBM API developed in Aim 1 of this study. This allows access to fully synced data from these systems. EEG data has simultaneously been recorded as well. The EEG system is from g.tec Inc. and the synchrony of the EEG and the API is provided through a trigger output from the amplifier of the EEG system.
Sample size calculation has not been performed as this experiment was an initial study for testing the feasibility of using brain-body fusion system to distinguish PD group from NT group. Instead, the target was to recruit 10 participants from each group.
Advertisements about the experiment was sent out to the community and the YMCA groups.
11 PD patients without deep brain stimulation (DBS) and 10 healthy NT participants were recruited. The criteria for recruiting PD participants were to be diagnosed with PD and not have DBS implants. The criteria for NT group recruitment was having no known neurological disorders. All the participants were consented prior to the experiment according to the IRB requirements and approval (#HU1718-185). The Montreal Cognitive Assessment (MOCA) was performed for PD patients prior to the experiment.
Further information about the time of diagnosis, initial symptoms, current symptoms, and medications have been collected from the PD group as well in order to be provided to neurologists along with the recorded data.
Due to technical problems and some data loss, the data from subjects NT10, PD10, and PD11 have been excluded. Therefore, the remaining of this dissertation is based on the data from 9 PD and 9 NT participants. Table 2 shows the information about the participants.  Finger tapping kinematics were recorded by the smart glove developed through this research and the other movements were recorded by the Mocap system. fNIRS and EEG neuroimaging optodes have been set on motor cortex area to capture the brain activity related to the movements. The placement of fNIRS optodes produces 18 channels of data.
Unfortunately, one of the light sources malfunctioned in the middle of the study, and two channels of fNIRS data were excluded from processing. 13   Each activity and how to perform them was explained to the participants in details prior to the experiment. The participants could practice before starting, but no data was collected during the practice time. The participants were asked to follow the visual cues on the screen in order to perform each task for 5 trials. Each trial consists of 10seconds activity followed by 10seconds rest. The participants were instructed to perform each task at their own pace. A green circle on the screen indicates participants should perform the task, and a blank white screen commands to stop and rest. Figure 21 shows the protocol of the experiment.

Data Analysis and Classification
This section explains the steps that have been taken in order to analyze the data recorded from PD patients and healthy NT group.

fNIRS Analysis
As mentioned in the previous chapter, the fNIRS system utilizes two different NIR wavelength light in order to measure the HbO2 and Hb based on the Beer-Lambert Law. The raw fNIRS data contains some environment noise and physiological noises such as heart rate and respiration rate which need to be removed from the data before calculating the HbO2 and Hb. This can be done by applying a band-pass filter with cutoff frequencies of 0.1 − 0.4Hz [6,7]. Therefore, a 4th order band-pass Butterworth filter has been applied to the raw data of fNIRS first. For calculating the HbO2 and Hb based on Beer-Lambert Law, it is important to define the extinction coefficient of hemoglobin. There are some small differences in the reported extinction coefficients of hemoglobin in the literature [8,9,10,11,12,13,14]. The reported values in [10] have been used in order to calculate the HbO2 and Hb as suggested by NIRx. Yet another parameter setting is DPF that takes into account the additional average distance light travels due to scattering beyond the linear distance between a source and receiver.
Values representative of reports described in the literature [15,16,17] have been chosen as follow: • DPF for WL1: 7.25 • DPF for WL2: 6.37 It deserves noting that the true value will depend on head shape, skull thickness, and properties of underlying tissues which is out of the scope of this study. After setting all the parameters and calculating the HbO2 and Hb concentrations, the upsampling and synching methods described earlier are applied to the fNIRS data.

EEG Analysis
The EEG signals were collected using the software BCI2000 [18] at the sampling rate of f s = 256Hz. The data analysis starts with detrending the data, or removal of the mean value or a linear trend from the signal. Detrending is done on the normalized signal using the z-score. The z-score is calculated by z = (X − µ)/σ , where z is the z-score, X is the value of the element, µ is the population mean, and σ is the standard deviation Epochs of the desired length are taken.
After detrending the signal, a Finite Impulse Response (FIR) band-pass filter with the order of 3 * f s and the cutoff frequencies of 1 and 99Hz have been applied on the data. In the next step, the power spectral density (PSD) of the signal has been calculated.
PSD provides information regarding when the average power is distributed as a function of frequency. It can be calculated as the Fourier transform of the autocorrelation function: After calculating the PSD for the EEG signal, the power in the frequency bands of Theta, Alpha, and Beta are calculated as follow: The power in these bands has been used to observe the brain activity in two groups of participants.
The spectrogram of the EEG data with a moving Gaussian window of length 1second with no overlap has been calculated as well. The time course of power modulation (TCPM) [19] has been calculated for each frequency band of Theta, Alpha, and Beta for the whole duration of the data.

Mocap Analysis
Mocap data has been recorded from 3 sensors. The locations of the sensors are as follow based on the activity.
• Sensor 1: Around Hand, or around foot • Sensor 2: Lower arm (below elbow), or lower leg (below the knee) • Sensor 3: Upper arm, or upper leg Each sensor provides three degrees of freedom of acceleration, gyroscope, and magnetometer. The data has been normalized and detrended as previously mentioned, and the normalized mocap data will be synched and upsampled based on the methods mentioned earlier, and the acceleration vector has been calculated from the three axes of it as follows:

Glove Analysis
Flex data was collected from the glove at a variable sampling rate of approximately 50Hz. A resampling method has been used to match the sampling frequency of the glove to the EEG, which is 256Hz. Standard preprocessing steps such as z-score normalization and detrending the data have been performed as well.

Classification
Support Vector Machine (SVM) is a supervised learning model which uses a nonlinear mapping to transform the original training data into a higher dimension. It searches for the linear optimal separating hyperplane in this new dimension (that is, a "decision boundary" separating the tuples of one class from another). With an appropriate nonlinear mapping to a sufficiently high dimension, data from two classes can always be separated by a hyperplane [20].
The general concept of the SVM is that the system uses a training dataset which can be a part of the original dataset that is labeled into different categories. Then, after the system is trained, it will be tested on the other part of the original dataset to predict the labels of the data, and by comparing the predicted labels and the original labels, the accuracy of the system can be found.
In this study, several learning algorithms have been implemented in the support vector machines. By applying the Lagrangian optimization theory to a linear support vector machine, and using the Kernel functions, one can classify datasets which are not linearly separable, while the nonlinear support vector machines retain the efficiency of finding linear decision surfaces while allowing them to be applied to non-linearly separable datasets. It is also possible to change the margins of the classifiers and change the complexity and accuracy of the systems. In general, large margins make the system less complex by letting the system make more errors and resulting in less accuracy. This can be achieved by changing a variable called Cost constant in the classifier models. Figure 22 shows a concept of the SVM and its supporting hyperplanes that divide two different classes and introduces the margin. Larger margin (lower Cost constants) would allow some data points between the decision surface and the supporting hyperplanes, which generally results in false classifications and reduced accuracy.  Table 4 shows different learning algorithms that have been used in this study with their complexity index which we will refer to them later in the results section.
The main goal of this study was to distinguish the PD and NT group based on the brain-body fused data. To achieve this goal, the data from all the devices have been fused together for classification. Due to the fact that this dataset is big data and performing the classification on this big data needs exceptional computing, the data size has reduced by averaging all the data in one-second intervals. Therefore, all the data points in a one-second window of the data have been averaged and treated as a single data point.
Considering the duration of the experiment for each activity is 100 seconds, the number Polynomial kernel k( x, y) = ( x · y + c) d 3 100 The averaging method is applied on the HbO2 data of all the 16 channels from fNIRS (total of 16 attributes), power spectrum of all the 13 channels of EEG for each Theta, Alpha, and Beta frequency band (total of 13 * 3 = 39 attributes), and the accel-eration data from Mocap or the flex data from the glove based on the activity (total of 1 attribute). Fusing all the data together provides a dataset with 16 + 39 + 1 = 56 attributes.
Participants were asked to perform each of the 8 activities for 5 trials. The first 3 trials of each activity have been selected in order to create the training dataset for the classifier, and the last 2 trials are left for testing the performance of the classifier. The classification between PD and NT group has been performed in two different scenarios: 1. Classifying the two groups for each individual activity. This means that all the data from all of the participants related to a specific task has been merged together in order to create the training and testing datasets, and all of the classifiers mentioned in Table 4 have been applied on them.
2. Classifying the two groups for all the activities together. This means that all the data from all the participants for all the activities have been merged together in order to create the training and testing datasets and the classifiers have been applied to the data.
The classifier performance results are usually shown in confusion matrices. Confusion matrices show how many of the samples in each class are predicted correctly, and how many of the samples in each class are misclassified. Table 5 shows a sample confusion matrix. In this study, the PD and NT groups have been assigned with the Positive label and Negative label, respectively. Three metrics based on the confusion matrix are reported in the results as follow: • Accuracy: The proportion of correct predictions to the total number of samples.
• Sensitivity: The proportion of actual positives that are correctly predicted (the percentage of PD samples that are correctly predicted) • Specificity: The proportion of actual negatives that are correctly predicted (the percentage of NT samples that are correctly predicted) List of References [2] "Nirscout system, nirx inc.:," https://nirx.net/.
[7] S. D. Power, A. Kushki, and T. Chau, "Towards a system-paced near-infrared spectroscopy brain-computer interface: differentiating prefrontal activity due to mental arithmetic and mental singing from the no-control state," Journal of neural engineering, vol. 8, no. 6, p. 066004, 2011.

CHAPTER 4 Results
This chapter presents the results of this study. Each section of the previous chapter introduced a part of this study and some experiments in order to validate that part. This chapter is organized as follow: •  This section concludes with the results of the classification to distinguish between PD and NT groups.

WearUp: Finger Tapping Test Results
This section presents the results of the preliminary analysis of the data collected when using the robotic hand and from healthy participants. Figure 23 shows the results of the peak detection algorithm on the data from the robotic hand. The red markers are the peaks detected by the algorithm. Since we are operating a robotic machine with defined angles and flexions, we observed that the amplitude [see Figure 23] and frequency [see Figure 25] of the finger tapping activity were almost identical at each consecutive tap. Figure 24 shows the results of the algorithm applied to the data recorded from a healthy participant, with the detected peaks marked in red. It is apparent that each finger tap is represented with one peak, and the variation of human movements can be seen by the magnitude of the peaks and the width (duration) of each finger tap.  After applying the peak detection algorithm, and finding the location of the peaks, we calculated the frequency of finger tapping by finding the number of samples between each two adjacent peaks and dividing the sampling rate with that difference as follow: Where f is the frequency in Hz, f s is the sampling rate, and N is the number of samples between two adjacent peaks: The consistency of the performance can be revealed by the results of peak detection and frequency calculation. Looking at the resistance value of the sensor at the peaks (the magnitude) reveals whether or not the subject has performed the finger tapping task in a similar way each time or if the position and the angle of the finger was different at each single finger tap. As an example, we can see that the first finger tap of the slow speed has a significantly lower peak compared to the other finger taps (see Figure 24). On the other hand, by looking at the frequency of finger tapping for fast or slow speed task, we can understand how consistently the subject pursued the task. A good measure of this consistency is the mean and variance of the finger tapping frequency. The mean of the frequencies reveals how fast or slow the subject has been performing the finger tapping task, and the variance reveals the magnitude of fluctuation in frequencies. The smaller variance means more similarities of the frequencies or more consistency in the speed of finger tapping. Table 6 shows the mean and variance (σ 2 ) of the finger tapping for the three rounds.
Due to the fact that the robotized hand is operated by a computer with specific parameters, a very small variation should be observed in the frequencies compared to the healthy subjects that show higher error because of hand micro-movements or human error.
As an example, Subject 1 has performed the fast finger tapping task at a higher speed in the first round (average frequency of 3.76 Hz) compared to the second and third round which the average frequency is 3.05 and 3.04Hz respectively. Regarding the variability of the participants, the variance (σ 2 ) provides the information. As an example, Subject 2 in the fast finger tapping of round 1 was not very consistent and had variability in performing the task (σ 2 is 0.149 which is slightly high). On the other hand, a good example of consistency with small variability is Subject 5 in the slow finger tapping of round 1, which the variance is 0.001. The robotized hand, as discussed, should have a steady speed in performing the task. Table 6 shows that the average frequency of finger tapping is consistent for different rounds as 0.654 and 0.327 for fast and slow finger tapping respectively. Also, the variation of the task is negligible in the range of 10 −5 for the robotized hand. Figure 25 shows the boxplot of the frequencies of the fast finger tapping from five participants and the robotic hand. All of the participants have been very consistent in performing the task but there is variability between subjects. On the other hand, the frequency of finger tapping by the robotic hand is consistent and there is no variation in performing the task, thus these results provide the validation of the WearUp glove's hardware design , textile design, and algorithmic model.

BBM API System Level Testing Results
In this section, the results of the experiments explained in the previous section are provided.

Mocap-fNIRS Synchrony Results
In this test, one pair of the fNIRS optodes and a single Mocap sensor were placed on the arm of the participant while the participant was asked to tap on the desk every 25seconds. After upsampling and synchronizing the data, a peak detection algorithm was applied in order to find the location of the peaks which represent the hand tappings.   It can be seen that the average time difference between Mocap and fNIRS data is in the range of milliseconds. Although the results show that these systems are not completely synchronized, considering the slow response of fNIRS which is in the range of a few seconds, makes this time difference negligible and acceptable.

Brain-Body Sensor Fusion Results
In this section, we present the results of the second objective of this part. As mentioned earlier, the data has been collected from 18 control participants, and the participants were asked to follow the visual cues in order to perform different tasks. After calculation of the HbO2 and Hb from the fNIRS data, and also calculating the acceleration from its three vectors, the goal is to test the performance of the developed system in recording the data and syncing the data. Figure 28 shows the HbO2 and accelerometer signals for the 20seconds of rest followed by the 10seconds of right leg activity.
The accelerometer signal is coming from the IMU sensor placed on the right foot, and the HbO2 is from the fNIRS channel set by placing the light source at C1 location on 10 − 20 map, and the detector on FC1. It can be seen from figure 28 that by starting the movement which can be observed by the accelerometer sensor, the level of HbO2 increases as well. Figure 29 shows the detection of the increase in HbO2 level for different activities of right leg movement, left-hand flipping, and both arms movements. It is clear from the figure that the level of the HbO2 increases by performing the movement activities, and it is happening contralateral to the moving side of the body. Figure 29: Hemodynamic maps of the brain based on different movement activities. The increase in the level of OxyHemoglobin is observed on the contralateral side of the brain from the movement on the motor cortex area.

PD vs. NT Experiment Results
This section presents the results of the experiment conducted On PD patients and NT participants. A brief representation of the results of each individual system is shown, and the section concludes with the results of classification between PD and NT group.

fNIRS Results
The changes of HbO2 levels are shown here. The HbO2 levels have been calculated and after upsampling and synchronizing the data, the changes of HbO2 level have been calculated based on the difference between the maximum of HbO2 level during the task period, and the minimum of HbO2 level during the rest period. The differences were averaged over all the 5 trials for each subject. Figure 30 shows the changes of HbO2 levels for one of the NT participants for each activity.

EEG Results
EEG data require neither upsampling nor synchronization. The PSD of the data for each activity and participant has been calculated in different frequency bands of Theta, Alpha, and Beta. Figures 31,32,33 show the PSD of each activity from one participant in the Theta, Alpha, and Beta band, respectively.

Mocap Results
The Mocap system records the triaxial data of accelerometer, gyroscope, and magnetometer. After the data is upsampled and synchronized, the acceleration vector is calculated from the three-axis accelerometer components to be used for the classification.

Glove Results
The data recorded from the flex sensor is upsampled, synchronized, detrended and normalized in order to be used for the classification. Figure 35 shows a cycle of rest period followed by finger tapping for a participant of the NT group (top panel) and PD group (bottom panel).

Classification Results
All the analyzed data described earlier have been fused together for classification in order to distinguish between the PD and NT group. 12 different SVM classifiers have been used in this study which are introduced in Table 4. The index of the classifiers will be used in this section. The classification has been performed in two different scenarios.
One is to distinguish PD and NT group based on each of the 8 activities separately, and the second is to distinguish PD and NT group based on all the data together. Table 7 reports the accuracy, sensitivity, and specificity of the 12 classifiers to distinguish PD and NT group based on each activity separately.

Activity Based Classification
It can be seen that for each activity, the PD and NT group can be distinguished by   It is clear that by using more complex classifier (higher index), the performance of the classifier regarding the accuracy, sensitivity, and specificity converges, which means the optimum performance. The highest accuracy of distinguishing between PD and NT group by using only fNIRS data is 81.23% with 83.57% and 78.89% sensitivity and specificity, respectively by using the SVM with index 12. Figure 37 shows the performance of all the 12 classifiers on the dataset containing only EEG data. The highest accuracy of distinguishing between PD and NT group by using only EEG data is 92.79% with 93.12% and 92.46% sensitivity and specificity, respectively. Figure 38 shows the performance of all the 12 classifiers on the dataset containing only fNIRS and EEG data.   The highest accuracy of distinguishing between PD and NT group by using all the data is 93.40% with 93.78% and 93.02% sensitivity and specificity, respectively.
It can be observed that adding more modalities to the dataset for the classifier increases the accuracy of distinguishing PD and NT group. This reveals the importance and critical role of each modality that has been used in this study.

CHAPTER 5 Conclusion & Discussion
This study is focused on fusing the brain neuroimaging and body motion data.
Body kinematics were recorded for both fine and gross motor tasks using a smart glove and Mocap system, respectively. fNIRS and EEG were the two non-invasive portable modalities that were used as brain neuroimaging tools.
To assess fine motor movements related to UPDRS tasks, there was a demand to develop a device capable of measuring fine motor movements precisely. WearUp glove was developed in order to meet the requirements. WearUP is an e-textile glove integrated with sensors on the fingers, which can monitor fine movements of the fingers.
The WearUp Glove has the potential to be used for remote assessments of movement disorders such as PD. The preliminary experiments with WearUP showed promising results. The most challenging part of this development was the design of the glove, which required many iterations. One particularly difficult design issue involved the identification of a stable stitching line which kept the sensors in the appropriate places on the finger joints while fitting well on the hand during movement.
To measure the gross movements of the body, a full body motion capture suit consisting of 17 IMU sensors has been utilized. The sensors were connected to dongles wirelessly in order to record the triaxial information of accelerometer, gyroscope, and magnetometer from each sensor. 3 sensors were used in this work covering each limb separately.
The main objective of this work was to establish a cyberinfrastructure to simultaneously collect the multi-channel data from a portable brain monitor (fNIRS) and a full-body motion capture system (Mocap). The project was aimed to assist the interventions of PD.
For this objective, the project outcomes are summarized in three parts: 1) Integration of Brain and Body Monitoring Systems: A data acquisition system that could collect multi-channel sensor signals from commercial devices -fNIRS brain monitor and inertia-based full-body motion capture system, was developed, validated, and tested. Each of these systems provides an individual separate data recording software, and creating a framework in order to record the data simultaneously from these two systems was the most fundamental challenge. Therefore, an API that reads the data from all the sensors in the systems, and saves it on separate files, all with the timestamps provided by the clock of the recording computer with milliseconds accuracy was developed. The data acquisition system was designed after a thorough investigation in the area of system integration and software engineering. The latest software is a oneclick interface taking care of connecting, initializing, and gathering the data from two complex systems.
2) Integration of Brain and Body Data: When the data from two systems are centralized and stored, they need to be analyzed for the measurement of how the body and brain are synchronized. The data arriving from both systems are inconsistent because the sensors are sampled at different speed. Signal processing algorithms to synchronize the data were developed in order to overcome this issue.
The performance of the acquisition system and developed algorithms was tested by conducting two sets of experiments. In the first set of experiments, the goal was to determine if generally the recorded data are synchronized or not. And the results of the first experiments show that the data is acceptably synced. Although there is a time delay between fNIRS and Mocap data based on the results of this test, considering the fact that the hemodynamic response of the brain usually have a delay of several seconds, few milliseconds delay in the recording of the Mocap and fNIRS data is acceptable and negligible.
The goal of the second set of experiments was to determine the performance of the developed system in real experimental situations and testify if the methods used in this study are able to detect the increase in HbO2 levels of hemodynamic response related to the movements of the body that can be detected by the Mocap system. The results of the experiment reveal that the body movement detected by the Mocap system is in synchrony with the brain activation demonstrated by the increase in the HbO2 level.
3) The last objective of this study was to perform an experiment on PD patients and NT group aiming to distinguish between PD and NT. The brain imaging systems used for this experiment were fNIRS and EEG. Body kinematics were measured by Mocap system and smart glove. 9 PD patients and 9 NT participants were recruited to perform 8 motor tasks on upper and lower limbs of both sides of the body. The data were recorded through the developed API and were synchronized using the developed algorithms. HbO2 from the fNIRS data, and PSD in the frequency bands of Theta, Alpha, and Beta were extracted from EEG data. The acceleration vector was calculated based on the triaxial accelerometer measures of the Mocap, and the flex sensor voltage was used from the smart glove. The data from all these modalities have been fused and averaged by a moving window of 1 second length. The final processed data were used for classification. The goal of classification was to distinguish PD and NT group.
12 different SVM classifiers were trained based on the first three trials of the data, and the performance of the classifier was tested on the last two trials. The classification was performed in two different scenarios of activity based, and the full data.
In activity based classification, the data from each activity were used separately to train and test the classifiers, and the results show that the PD and NT group can be distinguished with at least 85% accuracy for each activity.
Full data classification has been performed on four different scenarios: fNIRS only, EEG only, hybrid fNIRS and EEG, and all the fused data. The most optimum accuracy of distinguishing PD and NT group has been 81.23%, 92.79%, 92.27%, and 93.40% for the four scenarios, respectively.
The lower accuracy of the fNIRS only classification might be due to the nature of this data. fNIRS measures the changes of HbO2 levels which is carried by the blood. This is a hydraulic system and there are delays in transportation of blood. The delay of hemodynamic responses, that is based on changes of HbO2 level can be seen by calculating the cross-correlation of the fNIRS data and Mocap data. Cross-correlation is a measure of similarity of two series as a function of the displacement of one relative to the other. Therefore, the time of the peak in cross-correlation of two series shows the delay of the signal. Figure 40 shows the cross-correlation of fNIRS and Mocap data.
It can be seen that the peak of the cross-correlation is happening at 3-6 seconds. This means that the fNIRS HbO2 levels are having this specific delay. This delay in the response of fNIRS which is different within subjects prevents the classifier to find an optimum path for discriminating the two classes, which results in lower accuracies. Many studies have been done individually on measuring the kinematics of the body in people with movement disorders [1,2,3,4,5,6], or monitoring the brain activity in movement disorders or discriminating different activities based on the brain activity [7,8,9,10,11,12,13]. This study merges these separate studies together in order to provide clear insight about the synchrony between brain and body. Some studies have been done on measuring the synchrony of brain and body, but they usually utilize one modality of brain monitoring such as EEG or MEG, and one modality of body kinematic recording related to gross motor tasks or muscle contractions by electromyogram (EMG) [14,15,16,17,18,19,20,21,22,23,24,25].
This study is among few studies, if not the only study, in fusion of brain and body data with two different modalities of brain imaging and measuring both fine and gross movements. The promising results show the feasibility of using all different recording modalities.

Outlook
This dissertation demonstrated the research and development of a brain-body fusion system and evaluation tests of the system. Experimental results on the PD and NT group showed the feasibility of using this system to distinguish PD from NT groups. It is observed that the two groups could be distinguished by more than 85% accuracy for each individual activity. The results of classification for all the data demonstrated that each of the modalities adds more information to the study which is shown by the increase of the accuracy while more data was used for classification. A quantitative evaluation needs to be performed based on sample size calculations in order to generalize the results of this dissertation for distinguishing PD from the NT group.
Adding more information about the disease history of each individual PD patient to the analysis such as the time of diagnosis, medications, dosage of medications, and UPDRS score from neurologists, along with performing a longitudinal study can provide a measuring tool about the progress of the disease or effectiveness of the medication. As an example, adding the UPDRS scores for each patient along with the type and dosage of medication they are taking into the analysis, and performing a multi-class classification can distinguish each patient separately.
The results of multi-class classification accuracies could be merged in a visualization interface to be used by neurologists. Figure 41 shows the overall concept of the proposed idea of developing the visualization user interface. The purpose of the visualization interface is to provide a tool to monitor the brain activity and body kinematics at the same time while the patient is performing the tasks.
Currently, this happens by visual inspection of the body kinematics while the patients perform the tasks and neurologists observe and score the patients based on UPDRS criteria and make the decision for diagnosis or progression monitoring. Adding the processing algorithms and results to the visualization interface can be served as a supporting tool helpful in the diagnosis of PD, or monitoring of the disease progression. Figure 42 shows the first step toward developing such a visualization interface.
Each circle on the brain represents a channel of the fNIR data and color gradient shows the changes of HbO2 level in the channel. The avatar replicates the movements of the subject based on the chosen sensors during the initiating process of the visualization interface. Figure 42: The developed visualization interface as a first step toward providing a supporting tool for decision making.
More detailed analysis on the data and adding the results to this interface could be a supporting tool for decision making regarding the diagnosis of PD. • Abtahi, M., Amiri, A. M., Byrd, D., & Mankodiya, K. (2017). Hand Motion Detection in fNIRS Neuroimaging Data. In Healthcare (Vol. 5,No. 2,p. 20).

List of References
Multidisciplinary Digital Publishing Institute.