Aim 1. Our goal is to improve NASA teleoperation training efficiency by scientifically customizing remedial training based on the measured spatial abilities of individual astronauts. Astronaut robotics trainees vary significantly in their initial performance, ability, learning rate, and level of mastery. Because the process of training astronauts to be qualified robotics operators is so long and expensive, NASA needs tools to predict robotics performance prior to training. NASA’s existing “Aptitude for Robotics Test” (ART) had not been statistically validated. Using a logistic modeling approach we investigated how well an astronaut’s ART scores predicted their spatial performance in subsequent evaluation testing (either Shuttle PDRS or Generic Robotics Training). ART was not found to be a reliable predictor. Based on our data analysis, we proposed changes in ART performance metrics to improve the predictive power. These have now been implemented and are being used in the current round of astronaut candidate testing. We expect to re-evaluate the modified ART dataset in year 4. Also, as described in a paper recently submitted to J. Human Factors, we tested the mental rotation and perspective taking spatial abilities of 40 active astronauts who had completed at least one robotics training course. We found logistic regression models that predicted who would achieve the top score in qualification evaluations. The model predictions appear reliable enough to be used to customize regular and remedial training, but not to make career defining decisions. The models could not reliably predict who would completely fail, because so few did. We have proposed improvements in GRT scoring methodology that should improve prediction reliability. At JSC’s request, we are also evaluating whether GRT scores could be used to predict performance in later training (e.g. SSRMS), or under operational conditions on ISS.

Aim 2: Our second major objective is to perform experiments using a space telerobotics training simulator at MIT to quantify how a trainee’s individual spatial and manual control abilities, use of camera views, and hand controller reference frame impacts learning and final level of performance as both primary and secondary robotics operator. This aim was the main focus of our activity this year. We hypothesized that individual ability to manipulate the arm and integrate camera views correlates with 3 subcomponents of spatial intelligence: spatial visualization (SV), mental rotation (MR) and perspective taking (PT). In particular, PT (the ability to imagine an object from another viewpoint) was thought to be important for integrating camera views. This year we completed three different experiments using MIT’s Dynamic Skills Trainer, a virtual space telerobotic training system similar to that used at NASA JSC:

In the first experiment, 19 subjects were trained to manipulate a robotic arm using a pair of hand controllers in a virtual environment almost identical to that in NASA’s Basic Operational Robotic Instruction System (BORIS), used in NASA’s introductory “Generic Robotics Training” course. Over 18 “fly to” trials, the disparity between the arm's control frame and the cameras was varied between low (< 90 degrees) and high (> 90 degrees) conditions. We used the Cube Comparisons (CC) test to assess SV, the Vandenberg Mental Rotations Test (MRT) to assess MR, and the Purdue Spatial Visualization of Views Test (PSVT) and a Perspective Taking Ability (PTA) test to assess PT. We showed that subjects with high PSVT scores moved the arm more directly to the target and were better at maintaining the required clearance between the arm and obstacles, even without a direct camera view. The subjects' performance degraded under the high disparity condition.

Our second experiment addressed trainee performance as both primary and then secondary (monitoring) operator. Twenty subjects were trained to manipulate the arm during 6 trials in a BORIS environment and then acted as a secondary operator observing an additional 32 trials in an ISS-like environment. We recorded which of three display monitors the trainee was looking at. The MRT, PSVT, and PTA were used to assess spatial abilities. Though the primary operator task was slightly different than that used in Experiment 1, we prospectively confirmed many results of the first experiment. Subjects with high PTA scores took less time, moved the arm more directly to the target, and moved the arm more fluidly, especially under the high disparity condition. High scorers on the PSVT and PTA were better at maintaining required clearance. Low PTA scorers looked from monitor to map more often. Prior experience with the arm didn't significantly improve task performance, and performance as primary operator didn’t reliably predict performance as a secondary operator. However, subjects with high PSVT scores had better overall secondary operator performance and high PTA scorers were better at detecting problems before they occurred. These two experiments are the Master’s thesis of Ms. Z. Tomlinson, and have so far been presented in two conference abstracts and posters.

Aim 3: Our third major goal is to identify and develop new interfaces and tools to support future in-space and lunar surface teleoperation and teleoperation training. Our original 2007 plan was to develop an adjunct spatial situation display and a scheme for switching camera views using operator gestures. We plan to focus on bimanual control skill assessment this year, while acquiring the necessary tracking hardware and address gesture control or spatial situation displays during Year 4.

Aim 1: Completed logistic model analysis of spatial test and JSC-ART data on 40 NASA astronauts to predict GRT scores. Manuscript submitted to Journal of Human Factors. Results presented at NASA HRP-BIW and ASMA. Changes in ART scoring proposed and have been implemented by JSC for next round. Reevaluation planned for year 4. GRT scoring methodology changes suggested.

Aim 2: Modified MIT dynamic skills trainer, developed new BORIS and ISS SSRMS tasks, developed IRB protocols and completed 3 series of experiments (n= total 60) on effects of spatial skills on primary and secondary operator performance and gaze patterns during simulated teleoperation training. Preliminary results of first two experiments presented at NASA HRP-BIW (Feb 09). Complete results in ASMA (May 09) poster and abstract and Master’s Thesis (February 09). Articles in preparation.

Aim 3: Discussions with the Robotics Training Branch suggested that we should also focus our research attention on the control and display issues associated with their newest challenge: training crews to use the ISS arms to successfully grab and dock ISS logistics supply vehicles such as ATV and HTV, which may be slowly drifting. Although the task does require spatial skills, it particularly demands that the operator develop a high degree of skill in bimanual control, so the end of the robotic arm can follow a three dimensional arc, all the while maintaining proper angular alignment with the docking pin. Arm translations are controlled with the left hand, and rotations with the right, so the operator must be able to instinctively decompose a 6 DOF movement into the corresponding 3DOF tasks for each hand. (The task feels a bit like to learning to draw smooth lines using a children’s Etch-a-Sketch tablet, except using rate rather than position control, and with six degrees of freedom, not just two.) Some individuals acquire much more proficiency than others. High performance normally requires extensive training and sustained practice. Relatively little is known about acquisition and retention of asymmetric coordinated bimanual control skills, or the origins of individual differences. The bimanual control literature largely addresses human computer interface tasks, where the left and right hands work cooperatively but separately on different (usually 2 DOF) tasks. The USAF employs a Two Handed Coordination Test to screen pilot candidates. However in flying the right hand controls attitude (3 DOF), while the left hand controls throttle (1DOF). There are no fully validated tests of bimanual telerobotic skill. We plan to develop such a test this year. So far our project has largely addressed spatial skills, so investigating the motor control aspects will usefully broaden our scope. Assessments of ATV/HTV docking task, bimanual control literature, and bimanual control test methodologies are underway.

59th International Astronautics Congress, Abstract Book, October 2008. , Oct-2008

Aviat Space Environ Med. 2008 Mar;79(3):288-9. , Mar-2008

NASA Human Research Program Bioastronautics Investigators' Workshop, Abstract Book, February 2009. , Feb-2009

Aviat Space Environ Med. 2009 Mar;80(3):221. , Mar-2009

International Astronautical Congress, Paper IAC-08-B3.6, September 2008. , Sep-2008

Our work towards these objectives has been separated into three specific aims:

Aim 1. To improve NASA teleoperation training efficiency by scientifically customizing remedial training based on the measured spatial abilities of individual astronauts. We have examined whether NASA-JSC’s current Aptitude for Robotics Test (ART) predicts the need for remedial work in Generic Robotic Training (GRT) and Shuttle manipulator training (PDRS) or whether additional psychometric tests will sharpen performance predictions.

Based on data from 40 astronauts, we have found that a logistic model statistically described the relationship between a standardized Mental Rotation Test score and General Situation Awareness and Clearance category scores during Generic Robotics Training (GRT) and Payload Deployment and Retrieval System (PDRS) qualification evaluations. Because the current training approach minimizes the total number of poorly scoring performers by training everyone to competence, our model estimates whether an astronaut will show excellent performance in training versus average or worse performance. We have evaluated the logistic model as a predictor of training performance (using the current data set) using an ROC methodology and found prediction performance is significantly better than chance. Before NASA could use such a model in practice, it will have to attach a cost to making erroneous predictions and a value to accurate predictions.

We found that the ART, as currently implemented, is not a very reliable predictor of performance during GRT. Of the five ART tasks, only two tasks are strongly correlated with astronauts’ evaluation scores. However we were not able to establish that these task scores are reliable predictors of GRT training performance using a logistic model. Additional measures of performance (e.g., quantitative temporal, spatial, or smoothness scores such as task-completion time, RMS error, and time-derivative of control inputs, respectively) both during training and during the final evaluation arguable could improve our ability to predict both outstanding performance and the need for remedial training. The three ART tasks that did not clearly correlate with training performance could be modified by imposing task-time constraints on them.

Aim 2. To perform a series of experiments using the MIT Remote Manipulation System Simulator to quantify how a trainee’s individual spatial and manual control abilities, use of camera views and choice of hand controller reference frame impacts learning and final level of performance as primary operator. Secondary operator performance in a clearance detection and estimation task is assessed using a signal detection/situation awareness probe paradigm.

For these experiments at MIT we have re-created the BORIS training virtual environment used in GRT at JSC. Subjects performed as series of arm positioning tasks as the primary robotic operator under various visual conditions to test the role of spatial ability. We found that subjects with higher perspective-taking test scores tended to perform better in certain metrics of performance, such as showing smaller deviations from the ideal trajectory when positioning the robot arm and having fewer clearance violations during the trials, especially when the camera views do not provide direct estimates of the distance between robot arm and environment structure. These results suggest that training could be customized to emphasize different aspects of the task according an assessment of individual spatial abilities.

As expected, subjects’ performance was degraded, in terms of larger path deviations from the ideal trajectory, when the disparity between the camera and control frames of reference was greater than 90 degrees. However, there was no significant difference in task performance between subjects with low and high spatial test scores. There were no differences between the subjects in terms of their improvement in task performance. While the higher scoring subjects may grasp the spatial aspects of the task more quickly, all subjects may have similar difficulties learning the appropriate motor mappings to control the arm. This suggests that tests of individual ability in manual control or dexterity could also be useful predictors of robotic task performance.

Aim 3. To develop and evaluate two interactive interfaces for future in-space and lunar surface operations. Work on this specific Aim is scheduled to begin by Year 3.

Proposed work for Year 2

We will continue data collection at NASA-JSC for Aim 1 to reach the desired sample size. With a sufficiently large sample size, we may be able to test the reliability of our predictions using a split-half technique. We will also carry out a new analysis suggested by the NASA Astronaut Office of predicting "real-arm" training performance from the Generic Robotics Training performance.

Work on Aim 2 will continue with the completion of the second experiment investigating the role of spatial ability in secondary operator performance of monitoring telerobotic operations. Further experiments will be developed that will investigate control mode awareness, and monitoring performance when an end effector camera is in use. We will also refine the use of gaze tracking in these experiments to understand how visual information is utilized during operations.

Aim 1 - To improve NASA teleoperation training efficiency by scientifically customizing remedial training based on the measured spatial abilities of individual astronauts.

PROGRESS – Four MIT investigators took Generic Robotics Training at JSC. We collected additional data at JSC to bring our subject population to 40. Analyzed the correlations between spatial ability tests and ART, Generic Robotics Training and Shuttle arm training using a logistic regression technique. Demonstrated a technique to predict GRT performance from spatial tests scores. Made recommendations to Scott Hobaugh, Astronaut Office Robotics Branch Chief, and James Tinch, Robotic Branch Chief Engineer, on the efficacy of ART. Beginning analysis on suitability of GRT performance as a predictor of Shuttle or Space Station arm performance. Presented results at the 2008 AsMA scientific meeting.

Aim 2 - To perform a series of experiments using the MIT Remote Manipulation System Simulator to quantify how a trainee's individual spatial and manual control abilities, use of camera views and choice of hand controller reference frame impacts learning and final level of performance as primary or secondary operator.

PROGRESS – Developed a simulation of the BORIS virtual reality teaching environment used in GRT. Completed the experiments for Experiment 2.1 studying performance of a primary operator, including test scenarios and instructional materials. Completed data analysis and submitted an abstract for the 59th International Astronomical Congress, Glasgow, Scotland (Oct, 2008). Modified the task scenarios and instructions for Experiment 2.2 studying performance of a secondary operator. Developed a simple video data recording system to collect eye gaze information during the tasks. Data collection for this experiment started July 2008. Developed a demonstration version of the experiments for educational and outreach purposes.

Aim 3 - To develop and evaluate an improved spatial situation display and new camera control interaction techniques for future in-space and lunar surface operations. PROGRESS – No work scheduled during this year.

59th International Astronautics Congress, Abstract Book, Oct 2008. , Oct-2008

Aviat Space Environ Med. 2008 Mar;79(3):288-9. , Mar-2008

1. To develop tests of astronaut spatiomotor abilities that predict the need for remedial training or performance in final telerobotic qualification tests; and

2. To improve teleoperation training techniques and develop new teleoperator interfaces that improve the efficiency of teleoperation training and flight operations.

Specific aims are:

1. To improve NASA teleoperation training efficiency by scientifically customizing remedial training based on the measured spatial abilities of individual astronauts. We propose an experiment to determine whether NASA Johnson Space Centers (JSC) current Robotic Aptitude Assessment test predicts the need for remedial work in Generic Robotic Training and Shuttle PDRS manipulator training or whether, as we expect, additional psychometric testing will sharpen performance predictions.

2. To perform a series of experiments using the Massachusetts Institute of Technology (MIT) Remote Manipulation System Simulator to quantify how a trainees individual spatial and manual control abilities, use of camera views and choice of hand controller reference frame impacts learning and final level of performance as primary operator. Secondary operator performance in a clearance detection and estimation task is assessed using a signal detection/situation awareness probe paradigm.

3. To develop and evaluate two interactive interfaces for future in-space and lunar surface operations:

* An improved, user-controllable work area spatial situation display; and

* A new head-gesture controlled method for switching between camera views, thereby reducing or eliminating the requirement for multiple monitors in telerobotic workstations.

The short psychometric spatial ability test subjects we employ are sensitive to cognitive impairments and are candidates for Flight Medicine fitness-for-duty tests for astronaut telerobotic system operators. Our approach builds on evidence from our prior research that specific spatial abilities are correlated with teleoperation performance metrics.