Author(s) Reynolds, James V.; Smith, Craig L. Title Virtual environment training on mobile devices Publisher Monterey, California: Naval Postgraduate School Issue Date 2013-09 URL http://hdl.handle.
NAVAL POSTGRADUATE SCHOOL MONTEREY, CALIFORNIA THESIS VIRTUAL ENVIRONMENT TRAINING ON MOBILE DEVICES by James V. Reynolds Craig L.
THIS PAGE INTENTIONALLY LEFT BLANK
REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.
THIS PAGE INTENTIONALLY LEFT BLANK ii
Approved for public release; distribution is unlimited VIRTUAL ENVIRONMENT TRAINING ON MOBILE DEVICES James V. Reynolds Major, United States Marine Corps B.A., Bucknell University, 1996 M.S., University of Rhode Island, 1999 Craig L. Smith Major, United States Marine Corps B.S.
THIS PAGE INTENTIONALLY LEFT BLANK iv
ABSTRACT Over 100 million tablet computers have been sold in the last three years. They now have the computing power of a state-of-the-art laptop of just a few years ago. This computing power and market saturation allows them to become viable virtual environment (VE) trainers. Tablets have a different set of input modalities and user expectations, which need to be taken into careful consideration when a VE trainer is designed.
THIS PAGE INTENTIONALLY LEFT BLANK vi
TABLE OF CONTENTS I. INTRODUCTION ............................................................................................. 1 A. PROBLEM STATEMENT..................................................................... 1 B. MOTIVATION ....................................................................................... 2 C. RESEARCH QUESTIONS ................................................................... 5 D. ORGANIZATION OF THE THESIS ...................................................... 6 II.
C. D. E. F. G. H. I. J. K. a. CASE 1: ........................................................................ 30 b. CASE 2: ........................................................................ 30 4. Characteristics of the DVTE and Laptops ........................... 31 5. Physical Interaction with the DVTE / CAN ........................... 32 a. Input ............................................................................. 32 6. SAVT / MSAT ........................................................
1. C. D. Analysis of Likert Questions ................................................ 60 a. Question 1: Training with this Device on a Regular Basis Will Improve My Ability to Conduct CFF in the Field ....................................................................... 60 b. Question 2: It Was Difficult Navigating through the Device to Find the Appropriate Information While Completing the Tasks ................................................. 60 c.
E. F. OPEN ENDED QUESTIONS .............................................................. 67 DISCUSSION ..................................................................................... 69 1. Is a VE trainer on a tablet possible? .................................... 70 2. Is the “window to the world” paradigm seen as a valuable addition to VE training? ......................................... 70 3. Would military officers both trained and untrained in CFF see a value in VE tablet CFF training? .......
LIST OF FIGURES Figure 1. Figure 2. Figure 3. Figure 4. Figure 5. Figure 6. Figure 7. Figure 8. Figure 9. Depiction of “window to the world” using SAT-M .................................. 5 Photo of students using TSFO to practice CFF procedures (From United States Army Field Artillery School, 1989) ................................ 11 FOPCSim screen capture................................................................... 13 Marines using SAVT (From Bilbruck, 2009) .......................................
THIS PAGE INTENTIONALLY LEFT BLANK xii
LIST OF TABLES Table 1. Table 2. Table 3. Table 4. Table 5. Table 6. Table 7. Table 8. Table 9. Table 10. Table 11. Table A1. Table A2. Cognitive and specific knowledge / skills needed to perform CFF tasks (After McDonough & Strom, 2005) ............................................ 20 Psychomotor and sensory perceptual abilities needed to perform CFF tasks (After McDonough & Strom, 2005) ....................................
THIS PAGE INTENTIONALLY LEFT BLANK xiv
LIST OF ACRONYMS AND ABBREVIATIONS AO Air officer CAN Combined arms network CAS Close air support CFF Call for fire CFFT Call for fire trainer CLRF Common laser range finder COTS Commercial off-the-shelf GOTS Government off-the-shelf CTA Cognitive task analysis DAGR Defense advanced GPS receiver DoD Department of Defense DOS Disk operating system DVTE Deployable virtual training environment EWTGPAC Expeditionary warfare training group pacific FDC Fire direction center FiST Fir
HOB Height of burst HSV High speed vehicle ICM Improved conventional munitions IOC Initial operational capability ISMT-E Indoor simulated marksmanship trainer-enhanced ITS Individual training standards ITX Infantry training exercise IZLID Infrared zoom laser illuminator designator JFETS Joint fires and effects trainer system JFO Joint forward observer JSAF Joint semi-autonomous force JTAC Joint terminal attack controller LTD Laser target designator MCAGCC Marine Corps air ground c
RFMSS Range facility management support system SAT-M Supporting Arms Trainer-Mobile SAVT Supporting arms virtual trainer SDS Software delivery system SME Subject matter expert SOP Standard operating procedures T&R Training and readiness T/O Table of organization TACP Tactical air control party TSFO Training set fire observation TTECG Tactical training exercise control group TTPs Tactics techniques and procedures UAV Unmanned aerial vehicle USMC United States Marine Corps VE Virt
THIS PAGE INTENTIONALLY LEFT BLANK xviii
ACKNOWLEDGMENTS First and foremost, we would like to thank our spouses for their support and patience during this process. We would also like to acknowledge the Navy’s Modeling and Simulation Office, who provided key financial support to the project. To our thesis advisor Dr. Joseph Sullivan and our second reader Erik Johnson, thank you for your insights and encouragement.
THIS PAGE INTENTIONALLY LEFT BLANK xx
I. A. INTRODUCTION PROBLEM STATEMENT Military simulation training is not what it should be. It is often slow to adopt new technology. When innovations are adopted, frequently they are shoehorned into old paradigms, failing to maximize their advantages. This results in military simulation training that is not as effective as it could be, training opportunities are lost and expensive older simulators are run when a better cheaper option should be available.
work performed to produce CFF VE simulations for desktop / laptop systems. The development path was tailored towards the unique features and capabilities of tablet systems. The authors felt it was paramount that advances in technology be incorporated into the design process. After a functional prototype was created, we devised an experiment that compared the tablet system to an existing desktop / laptop CFF VE training system.
Department of Defense (DoD) must look towards easily distributed simulation as a viable alternative to live fire training (Deputy Commandant for Combat Development and Integration, 2012). United States Marine Corps (USMC) 2012 Science and Technology Plan identifies a critical training and education gap in training science and technology objective number six: Warrior simulation: Marines need to train as they would fight as small units, particularly for dismounted operations.
for fire skills. Upon completion of combat tours, where units are tasked to perform missions outside their primary skill sets, servicemen return with significant atrophy in their skills. Existing CFF training simulations include deployable options, which have been proven to adequately address cognitive skill retention in CFF tasks (McDonough & Strom, 2005).
Figure 1. Depiction of “window to the world” using SAT-M When one adds a virtual environment with high fidelity graphics and sound, we believe it will deliver affordable, portable, and quality simulation training. Throughout this effort, we will strive to outline and define the way forward for future development in the area of realistic virtual training C.
This analysis provides a framework for the progression of virtual training over the spectrum of desktop, laptop, and mobile tablet PC devices. In the process we answer the following questions: D. 1. Is a VE trainer on a tablet possible? 2. Is the “Window to the world” paradigm seen as a valuable addition to VE training? 3. Would military officers trained in CFF see a value in VE tablet CFF training? 4. Would military officers untrained in CFF see a value in VE tablet CFF training? 5.
Chapter VI, experiment outlines the methods and research process followed. Chapter VII, results, answers research questions one through four. Chapter VIII, conclusions, describes the authors’ findings. Chapter IX, future work, gives an overview of the way ahead for follow on research and development.
THIS PAGE INTENTIONALLY LEFT BLANK 8
II. A. BACKGROUND INTRODUCTION In 2010, Ben Brown conducted an in-depth review of simulation-training lineage, which he traced back to the very earliest versions of “serious games”. Examples of these early simulations include chess, Wei Ch’I, and Chaturanga. Brown’s discussion establishes the storied history of the relationship between simulations and military training (Brown, 2010).
maintenance. Portable / deployable simulations are found on laptop and portable devices, their greatest advantage is availability and convenience for the user, but without support staff they are only as good as the software running on them. The following are examples of these different CFF training technologies. They are listed in chronological order based on earliest found reference to their use in the DoD. 1. M32 Sub-caliber Mortar Trainer, ca.
facilities, information technology services and contractor support (Headquarters Department of the Army, 1991). It is unknown if any of these systems are presently in use. Figure 2. 4. Photo of students using TSFO to practice CFF procedures (From United States Army Field Artillery School, 1989) MiniTSFO, ca. 1985 MiniTSFO was a DOS-based PC simulation developed by Captain Bill Erwin as a research project that was then incorporated at West Point for cadet artillery fires training.
integrated with the computer simulation for training. Personnel are also required to maintain and manage the equipment. (Program Manager Training Systems, 2013). 6. Forward Observer Training Simulator, ca. 1998 A computer based classroom training simulator, FOTS is primarily used by the Navy and Marine Corps for introductory school house CFF training. The system requires instructor support as well as facilities and personnel to maintain it (Naval Air Systems Command, Training Systems Division, 1998). 7.
Figure 3. 8. FOPCSim screen capture Joint Fires and Effects Trainer System, ca. 2003 JFETS is an immersive training simulation for the training and rehearsal of nearly all aspects of indirect fire control procedures. According to General Maples, the chief of field artillery in 2003, JFETS introduces highly realistic conditions and situations that add realism to the virtual environment. It is quite large and requires permanent facilities and contractor support (Maples, 2003). 9.
10. Call for Fire Trainer, ca. 2005 CFFT replaced GUARDFIST as the primary Army CFF simulation-training system. It is a classroom installed system that incorporates many of the tools used by joint forward observers. As many as 30 students can train on the system simultaneously. The design of the system necessitates a classroom environment with facilities and personnel (Mitchell, 2005). 11. FOPCSim 2, ca.
13. Supporting Arms Virtual Trainer, ca. 2009 The SAVT began as a non-program of record in the Marine Corps and Navy with the name Multi-purpose Supporting Arms Trainer (MSAT). In Figure 4 we see joint terminal attack controllers (JTAC) and JFOs training on SAVT utilizing its fully integrated real world equipment suite in an immersive virtual environment. The environment is projected onto a 15’ high by 10’ radius dome.
for Simulation Training and Instrumentation 2013 catalog lists four simulations that could be used for CFF procedural training and rehearsal. Presently, all virtual environment CFF procedure trainers fall within the spectrum of high-end large scale classroom facilities requiring significant additional resources, through deployable laptop based simulations, requiring a small degree of additional resources.
C. PREVIOUS WORK In 2002, David Brannon and Michael Villandre investigated the potential for a computer based CFF procedures trainer. Their efforts led to the development of FOPCSim. As previously described, FOPCSim was a proof of concept which showed a computer simulation could effectively reproduce the tasks required of a JFO. A thorough cognitive task analysis was conducted and the work established that many aspects of CFF procedure training can be trained inside a PC VE.
A significant finding of this work showed that a VE training simulation could be used to maintain certain perishable skills. Both sets of researchers identified a training shortfall resulting from the lack of available simulation capability. Their proposed solutions and designs were based on the technologies of their times.
III. TASK ANALYSIS Our task analysis begins where McDonough and Strom’s ended in 2005. As previously discussed, they conducted a HARs absence presence assessment as part of their research. Their assessment revealed that psychomotor and sensory perceptual tasks are not well replicated within a desktop / laptop VE CFF simulation (McDonough & Strom, 2005). A HARs assessment compares the execution of a real world task to the execution of that task in a VE.
A. HUMAN ABILITY REQUIREMENTS REVIEW In 2005, McDonough and Strom used the HARs taxonomy to identify 27 skills required to perform CFF tasks. Of the 27 skills, 12 were identified as cognitive skills and are listed in the top portion of Table 1.
with a 1.4 GHz dual core A6X processor, and 1 GB of RAM, making the tablet as least as powerful as the PowerBook G4 (Shimpi, 2012). If it was possible to create a CFF VE training simulation that runs on the PowerBook G4, then it is certainly possible to create one that will run on the current generation of tablet computers. Therefore, we conclude that the cognitive component of the CFF VE can continue to be replicated on modern tablet computers while maintaining the same level of training efficacy.
We were able to narrow the field of human abilities as they relate to simulated task mapping differences between desktop / laptop systems and tablet systems by focusing on psychomotor tasks. Our analysis focuses on the modality differences between desktop / laptop and tablet systems, and their ability to train psychomotor skills. Tablet devices have a unique input control methodology, using accelerometers and gyroscopes to capture movement.
CFF Human Abilitiy Requirements assessment Comparison Absense / Presence Test Human abilities Call for Fire Task Analysis (Tasks) 1.1.1 Utilize GPS. 1.1.2 Utilize Map and Compass. 1.1.3 Utilize available tanks sights or laser range equipment for resection. 3.2.3.2 Polar/Laser Polar 3.2.3.3 Shift from Known Point: 3.2.6 Send-Method of Fire and Control: 3.2.12 Conduct-Spottings: 3.2.12.1 Height of Burst 3.2.12.2 Range 3.2.12.
CFF Human Abilitiy Requirements assessment Comparison Absense / Presence Test Human abilities Call for Fire Task Analysis (Tasks) 1.1.1 Utilize GPS. 1.1.2 Utilize Map and Compass. 1.1.3 Utilize available tanks sights or laser range equipment for resection. 3.2.3.2 Polar/Laser Polar 3.2.3.3 Shift from Known Point: 3.2.6 Send-Method of Fire and Control: 3.2.12 Conduct-Spottings: 3.2.12.1 Height of Burst 3.2.12.2 Range 3.2.12.
complete a cross walk of the CFF tasks in the two publications to ensure that there have not been any significant revisions which would alter the previous task analysis or the HARs assessment. This comparison was limited to those tasks requiring psychomotor abilities as we have previously established that the remaining human abilities can be easily replicated (cognitive and specific knowledge / skills) or require special equipment (sensory / perceptual). The authors’ review of the FM 6–30 and ATP 3–09.
THIS PAGE INTENTIONALLY LEFT BLANK 26
IV. A. REQUIREMENTS OVERVIEW Before the requirements for SAT-M can be derived there needs to be an understanding of how it fits into the overall CFF simulation-training continuum. As discussed in Chapter II, currently the two most commonly used CFF simulators in the USMC are the SAVT and the software suite on the DVTE. In order to place the simulators within a greater context, use cases for the three systems are presented. These demonstrate the niches that each of the three simulators fill.
increasing or decrease difficulty as appropriate. He is also able to recall their past missions to look for trends, allowing him to focus the training on those areas the JFOs find most difficult. b. Case 2 Corporal Doe has been selected by his battalion to go to the JFO Course. In preparation for the course he has taken the MarineNet classes on CFF, received face-to-face instruction from one of the battalion’s JTACs, and downloaded SAT-M onto his roommate’s tablet.
of course multi-touch enabled touchscreens. This allows entirely new paradigms for interacting with the device. While SAT-M does not take a revolutionary approach to the user-device interface; it does try to take advantage of some of these inputs. Physically moving the tablet to change ones view, as described in the W2W, is one way. Modern tablet multi-touch enabled touchscreens allow for parallel inputs, more than one icon can be activated at a time.
hub to download their applications. They also expect the hub to indicate when their software is out of date, and provide those updates when prompted. This follows a centralized control structure, making it easy to push out software changes to the user. Where it falls short is in the distribution of custom made scenarios and environments. Some tablet operating systems make it extremely difficult to transfer custom made content from one tablet to another. 3. DVTE a.
the Marines to get practice in CFF mission planning and execution. He has set up a classroom in Battalion spaces with a squads worth of DVTE laptops. The Marines are running a scenario set in Twentynine Palms. They could be executing the mission planning on the laptops; but in this case the JTAC wants them to do it on real maps, using mapping pens and protractors, as they would do it if called on to conduct the mission in combat. As he is the only instructor, the JTAC created a simple scenario.
There are other barriers to the use of the DVTE, especially in battalion spaces. It is time consuming to set up and network the laptops, they take up a decent amount of space, and they are vulnerable to theft. 5. Physical Interaction with the DVTE / CAN There are some general assumptions about how the DVTE laptops are operated.
SAVT and playing the role of the aircraft. A TACP instructor is taking notes and evaluating the students’ performances. The three students cycle through the positions, executing slightly different missions each time. b. CASE 2 In July 2013, Jack Gavin, SAVT operator at Marine Corps Base Twentynine Palms, provided the following case study.
integrated real-world equipment suite. There are only nine of SAVTs in the DoD. They require a highly trained operator, one who is not only technically proficient with running the simulator but also tactically proficient with the mission sets. The SAVT is scheduled in the range facility management support system (RFMSS) at least 96 hours prior to the training event. The 96 hour scheduling requirement inhibits spontaneity; units that have a sudden opening in their schedule cannot take advantage of it.
MSAT/SAVT DVTE Location Only Nine in the DoD Battalion spaces Simulation center Personnel Requirements Dedicated operator with extensive tactical experience, either a former naval aviator or JTAC Availability Scheduled at least 96 156 Full hours in advance distributed active duty units Mobility No SAT-M / Anywhere If the laptops are User networked together there is usually need for technical support, typically provided by the battle sim centers Networkability Once, in 2012, as a Yes Yes proof of
D. SUMMARY OF CAPABILITIES Conceptually, SAT-M will perform much of the same functionality as FOPCSim 2. It will require a robust mission planning capability, embrace the inputs available on a tablet, and provide as much feedback as can be usefully incorporated without inhibiting the training processes. Ideally it will allow an untrained Marine to learn how to execute CFF on their own while not developing improper habit patterns. Table 6 outlines key software capabilities.
4. The SAT-M simulation shall replicate both enemy and friendly forces including tanks, trucks, personnel carriers, command and control vehicles, reconnaissance vehicles, forward area air defense weapons, dismounted infantry with their associated weapons, mortars, artillery and rockets. 5. SAT-M shall permit users to design new scenarios and revise existing scenarios. 6. SAT-M shall provide the capability to generate new scenarios for the ultimate purpose of mission rehearsal. 7.
2007, NAVMC 3500.42A, Tactical Air Control Party Training and Readiness Manual dated 8 October 2008. 11. The SAT-M shall replicate laser range finder / designator equipment (e.g., GLTD, PLDR) to include target observation, fixed and moving target tracking skills. 12. The SAT-M shall simulate shell bursts to include sound effects of the required projectiles, anywhere in the target area with an observer-target distance of six kilometers (threshold) or 12 kilometers (objective). 13.
22. The SAT-M shall delay the distribution of rounds by10 seconds between subsequent volleys for multiple round missions. 23. The SAT-M shall simulate time of flight of both low and high angle fire missions. The user may select a compressed time of flight option upon scenario selection. 24. The SAT-M will include full function simulation of the following equipment with the latest technology: binoculars, compass with mils and degrees, PLDR, IZLID, thermals, DAGR, Vector 21b and PRC117.
h. The instructor shall be able to observer the student’s current and past missions, as well pertinent data such as round accuracy, transmission errors, and recommendations. 31. The SAT-M shall compute "did-hit" grid location and HOB for each weapon and mean point of impact and HOB for each fire mission. 32. The SAT-M shall perform all known and future types of fire missions. 33. The SAT-M shall provide the functions needed to initialize and control the training exercise.
g. 38. F. SAT-M will have a virtual clipboard where the user can construct and record their CFF missions, mark round impacts, and target numbers. SAT-M will provide mission feedback. a. SAT-M will provide real-time prompting, dependent on user set tutoring level, to assist users who are having trouble with mission execution. b. SAT-M will give end of mission feedback based of analysis of expert behavior, and what they would have done in a similar situation. NONFUNCTIONAL REQUIREMENTS 1. 2.
4. G. b. SAT-M shall replicate operational equipment platforms when practical to provide training simulation. c. In accordance with DoD Directive 5000.59 all systems currently under development shall be compliant with HLA. d. SAT-M shall realistically replicate all subsystem sound effects, as well as inter-subsystem communication. e. Subsystem sound effects shall be in proportion to that of the actual weapon operations. f.
4. H. I. J. Graphical user interface (GUI) input for user action CONFIGURATION MODULE 1. Specify types, sizes, and location of targets 2. Stationary and moving targets (future) 3. Choose different terrain sets 4. Choose different observation post locations 5. Choose lensatic or M2 compass (degrees or mils) 6. Allow entry to configuration module during run time VIEW MANAGER MODULE 1. Binocular view 2. M2 or lensatic compass view 3. Target designator view 4. Thermal view 5.
5. K. c. HE / Variable time d. White phosphorus (WP) e. WP M825 f. Improved conventional munitions (ICM) g. Illumination Enter subsequent corrections a. Left b. Right c. Add d. Drop e. Up f. Down 6. Enter observer-target (OT) direction 7. End the current mission 8. Enter refinements 9. Establish known points 10. Utilize standard operating procedures (SOP) for immediate missions 11. Allow for sequential viewing of targets AFTER ACTION REVIEW 1. 2.
V. A. SYSTEM DEVELOPMENT BACKGROUND Examining SAT-M through the lens of the model-view-controller (MVC) design pattern was the first step in developing our application. We used the design pattern to explore the differences and similarities between desktop / laptop and tablet VE trainers. This helped us determine where to focus our limited resources and therefore maximize development efforts. Throughout the process we leveraged validated CFF VEs, primarily ObserverSim. 1.
SAT-M was initially conceived to run on both Android and iOS devices, thereby including both tablets and smartphones. Utilizing the MVC design pattern allows for code reuse, as only the portion of the software that interacts directly with the system would need to be changed. What follows is a discussion of the model and view aspects of the MVC pattern as it pertained to our development effort.
Our effort started by mapping the mouse and keyboard inputs of ObserverSim to one of the input modalities of the tablets. At our disposal were the multi-touch enabled touchscreen, accelerometers, and gyroscopes. From conception we knew that we wanted to have the user’s perspective controlled by the accelerometers and gyroscopes. Adding a single finger swiping interface to control the view was discussed but never implemented.
Figure 6. Screen capture of SAT-M’s vector 21b view In only one instance does ObserverSim use the right mouse button, and this for gathering range information with the Vector 21bs. In ObserverSim, if the both the left and right mouse button are pressed while in Vector 21b mode heading and distance to the object under the “pointing circle” is displayed. In Figure 6 the “pointing circle” is just below the technical vehicle.
Mapping the keyboard from ObserverSim to SAT-M was more difficult than mapping the mouse. In the interest of limited development time, we chose to use drop down menus, rather than include a fully functional keyboard. This resulted in the elimination of user controlled walking motion. In ObserverSim pressing W, A, S or D moves the user forward, left, right or backwards respectively.
background creates a region of the screen where the user’s only interaction is tool selection. This prevents the user from accidently sending another command if they missed the desired tool. For example, if the interface allowed for finger swiping to change perspective the system might infer a missed tool touch as a finger swipe, changing the viewer’s perspective and potentially disorienting them. Figure 7.
Figure 8. B. Screen capture of SAT-M’s naked eye view INTERFACE DESIGN STUDY An interface design study was performed to facilitate user interface development. The intent was to create an effective user interface for the SAT-M, and to include scenario election, mission execution and mission planning. Mock up screens were created in HTML. These allowed a user to flow through a mission, starting with the creation of a user profile.
C. OPERATING SYSTEM AND HARDWARE SELECTION At the time SAT-M development began, the two dominant mobile operating systems were iOS and Android. To ensure our program would reach the widest audience, we decided to develop software for both. SAT-M is not dependent on cellular network access, which simplified our platform choices. A 3rd generation iPad with 16 GBs of internal flash storage, the least expensive and most up to date model available at the time, represented the iOS platform.
perspective (vis á vis W2W) in a CFF VE had not been done before, numerous tech-demonstrations were created to validate the idea. Once both the Asus and the iPad satisfactorily demonstrated W2W, the user interface was discussed and planned. The HTML interface designed and tested previously was demonstrated to the development team, who then implemented the button graphics and logic. The devices to be simulated in the software were discussed with the team as well as the general CFF process.
• ObserverSim had a fully functional DAGR. The buttons on the virtual DAGR functioned as they would on a real DAGR. SAT-M used a static screen shot of the DAGR’s present position screen. • In developing the scenario for the experiment in ObserverSim it was not possible for the authors to precisely select the set of equipment they wanted to have available to the user. As SAT-M was developed from scratch only what was appropriate to the experiment’s tasks was presented to the user.
VI. A. EXPERIMENT BACKGROUND McDonough and Strom, in their work on FOPCSim 2, showed that a PC based VE CFF trainer can improve performance. Though their results were constrained by not being able to conduct a graded live fire event, there was enough evidence to show that the software they developed did indeed improve student performance. In essence SAT-M is an updated version of FOPCSim. It brings the simulator to tablet systems while updating it to reflect eight years of technology advancement.
or had been designated by their commanding officer (CO) to conduct CFF. There were exceptions to the classification. In a number of cases USMC weapons platoon commanders were classified as trained. A weapons platoon commander is the leader of company’s FiST, and would have extensive on the job training. In another case, an individual had had extensive CFF experience over a decade ago and none since, was classified as untrained. 2. Apparatus and Location a.
consisting of 155 mm howitzers. Additional differences between SAT-M and ObserverSim are discussed in Chapter V, section F. 4. Procedures a. Tasks Tasks conducted were derived from the Brannon and Villandre CTA produced in 2002 (the full text of the CTA is on pages 17 through 42 of Brannon and Villandre).
• Execute Protocol “B”, which is identical to Protocol “A” except the participant switches device, and the Likert scale and open ended questionnaire has four additional questions that directly ask the participant about device preference, as well as a final open ended question. • Complete a demographic questionnaire. Examples of the protocols and questionnaires can be found in appendix B, experimental design details. b. Conditions The experiment was a two by two cross-over design as shown in Table 7.
VII. A. RESULTS GENERAL The overarching goal of the experiment was to determine how viable the input modalities of the tablet system are when compared to the existing standard set by desktop / laptop systems. However, as the two platforms were running different software it is possible that platform preference was due to the software and not the hardware.
1. Analysis of Likert Questions a. Question 1: Training with this Device on a Regular Basis Will Improve My Ability to Conduct CFF in the Field With a p-value less than 0.002 the participants’ responses provided a greater indication that the tablet system will improve their ability to conduct CFF in the field when compared to laptop system. b. Question 2: It Was Difficult Navigating through the Device to Find the Appropriate Information While Completing the Tasks With a two-tailed p-value of 0.
f. Question 6: Having This Software Available at My Unit Would Improve My Units Ability to Perform Their Mission With a two-tailed p-value of 0.051, there is no indication of system preference. g. Question 7: It Was Hard to Understand what the Buttons Did With a two-tailed p-value of 0.24, there is no indication of system preference. h. Question 8: The 3D View Interface Was Intuitive With a two-tailed p-value of less than 0.
l. Summation Redundancy of Likert Questions, Eliminating Four sets of two Likert scale questions were similar to each other. For example, Q8: The 3D view interface was intuitive, and Q10: The overall interface is intuitive, are in essence asking the same thing. To prevent this and similar redundant questions from overly influential the results, the average of the redundant questions were used in the calculation. The similar questions are Q2 and Q5, Q3 and Q9, Q4 and Q7, and Q8 and Q10.
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 All Eliminate Redundant Questions Table 8. 3. n 13 22 21 18 22 11 24 21 23 23 30 29 Summed singed ranks Tablet Laptop System System 2 tailed p-value 91 0 0.0013 180.5 72.5 0.0794 215.5 15.5 0.0005 103.5 67.5 0.4362 164.5 88.5 0.2168 55 11 0.0511 191 109 0.2388 173.5 57.5 0.0392 241 35 0.0015 223.5 52.5 0.0083 383.5 81.5 0.0019 371 64 0.
numbered 11 through 14. The direct questions had the participant specifically state a preference between the laptop and the tablet systems. The questions allowed the authors to directly ask for a preference, and in the case of questions 12 and 13 to a limited degree control for differences between the two systems. The answers were analyzed using a sign test. Table 9 is a summary of the results and analysis. 1. Analysis of Direct Questions a.
answered tablet or laptop to the question. The right most column, p-value, is the probability of the values in the second and third column if the chance of either being chosen is 50 percent. In Question 11, one of the participants had no preference; hence the sum of the laptop system and tablet system columns is 31 instead of 32. Tablet Laptop Question System System p-value 11 24 7 0.003327 12 26 6 0.000535 13 27 5 0.000113 14 23 9 0.020062 Table 9. 3.
the virtual environment are the same”. Both training and order have a statistically significant effect on the participants’ answers to Q3. If the participant was untrained or used the laptop first they gave the tablet system a higher score than the laptop system. Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Summed Redundancy Removed Table 10. 2. Two Tailed P-Values Training Order 0.5320 1.0000 0.6675 0.6674 0.0491 0.0469 0.1763 0.8246 0.6111 0.1797 0.0694 0.4800 0.1559 0.6399 0.6004 0.2913 0.2458 0.6457 0.1560 0.
E. OPEN ENDED QUESTIONS The open-ended questions were phrased to allow participants the opportunity to express what they felt was most pertinent from their experience with the systems. As expected there were a variety of answers. Some related directly to certain features of the software, for example “Compass should have metal filament that lined over radial direction to aid in giving accurate report”.
wire. To get a good bearing with the tablet system the user needs to steady the system gyroscopes and accelerometers. Although the Vector 21b and compass are not particularly challenging to use, it is harder than the laptop system makes it appear, whereas the tablet system replicates some of the real world motor skills required to conduct the task.
trainee to get out of the classroom and practice in more realistic conditions, “You could take it outside put soldiers in full body armor & simulate a CFF w/out the range”. A significant number of comments discussed how to make both the tablet system and laptop system better. Voice recognition was the number one desired feature for both devices, allowing the VE user to speak the CFF, as they would in the real world, instead of filling out forms.
1. Is a VE trainer on a tablet possible? 2. Is the “Window to the world” paradigm seen as a valuable addition to VE training? 3. Would military officers trained in CFF see a value in VE tablet CFF training? 4. Would military officers untrained in CFF see a value in VE tablet CFF training? 1. Is a VE trainer on a tablet possible? The development effort shows that it is possible to create a tablet VE training simulation.
instructions. Due to these limitations, the HMD is only worn in the final portion of the DVTE mission, when there is little need to double check notes and finger placement. W2W does not have any of these issues; the multi-touch screen is both the user’s view of the world and interface. 3.
• System preference had nothing to do with software or system fidelity • System preference was influenced by how the tablet system reduces extraneous cognitive load, allowing the participants to focus their mental efforts on executing the mission and not fighting the interface 72
VIII. CONCLUSION A. GENERAL OBSERVATIONS There were three areas that the authors explored in this research. We looked at the software differences between SAT-M and ObserverSim, the use of multi-touch touchscreens as an input device versus a mouse and key board, and we explored the use of the W2W as a way to train psychomotor skills.
• Figure 9.
C. LIMITATIONS SAT-M as built is not production ready, and it will take significant effort to make it so. As reported in Chapter VII there is the possibility of confounding in the experiment. Like any military training, improper instruction leads to negative training, SAT-M needs to be able to provide proper instruction for those times it is used by an untrained user away from an instructor. Tablet systems are not an appropriate solution for every training situation.
THIS PAGE INTENTIONALLY LEFT BLANK 76
IX. A. FUTURE WORK IMPROVING SAT-M TRAINING SOFTWARE 1. CFF SAT-M requires further development before it can be introduced for training. Currently it has enough functionality to execute the experiment described in this thesis and some of this apparent functionality is just a façade. For example, when the user brings up the DAGR, a still image appears with a hard coded current location. If the user could move around in the virtual world the DAGR would soon give an invalid location.
• The mission planning should forgo the drop down menus and auto completion, requiring the user to remember or record the pertinent information required to create a six part CFF transmission. The mission planning should also allow the user to execute both grid and polar missions, and make adjustments. • At this level of development SAT-M does not require more than one scenario, as long as that scenario provides enough diversity to allow for multiple training missions.
• The mission planning capability should encompass the complete set of CFF missions to include continuous illumination, immediate suppression and suppression of enemy air defenses. c. • Tier Three, Individual Training The intelligent tutoring system is critical for allowing SAT-M to operate as a standalone VE trainer. A great deal of work needs to done in this area to ensure that the correct information is being collected and relayed back to the user in a useful format.
center (FDC), greatly increase the immersion and transfer of training. Learning to think before speaking and proper communications cadence are skills all Marines must master. b. Map Data Downloaded of the Internet Multiple technology companies provide high quality satellite imagery and elevation data over the Internet. Google and Apple are examples of two such companies.
• Does standing and using W2W keep the user attention for longer than sitting and doing the same tasks with a mouse and keyboard? • Does W2W appreciably improve a user’s psychomotor skills? Is there a measurable difference between using W2W as an input and using a mouse and a keyboard? C. NEW PLATFORM The idea behind developing VE training applications for tablets was conceived two years prior to the completion of this thesis, when the authors first arrived at NPS in the summer of 2011.
THIS PAGE INTENTIONALLY LEFT BLANK 82
APPENDIX A. INTERFACE DESIGN TESTING A. BACKGROUND SAT-M was heavily influenced by the validated processes used to produce earlier CFF VE. The unique input modalities and user expectations of tablet systems were taken into account during the design process. We stared with the interface design due to how multi-touch enabled touchscreens change the way users interact with the device.
functional system, two(2) through opinion without a fully functional system, and one(3) with our prototype. For the purposes of this project we have evaluated three of the six original success criteria. SAT-M interface will be successful if it achieves any two of the threshold criteria outlined in the table below. The interface will be highly successful if it meets any of the two objective criteria included in Table A1.
and everyone else. JTAC’s and JFO’s characteristics vary from service to service, so for the purposes of the study conducted we focus on United States Marine Corps (USMC) eligibility requirements. • JTAC. May be a winged aviator, or ground combat arms officer, or combat arms staff non-commissioned officer (E-6 and above). • JFO.
3. Procedures a. Tasks The testing participants were given five minutes to explore the prototype. If they felt that five minutes was not enough time to get comfortable with the system they were given five more minutes. None of the participants desired the extra five minutes. After getting familiar with the system they were instructed to complete four tasks. The tasks were chosen based on a task analysis conducted during an earlier project for CS3004 coursework.
Survey Data Question 1 2 3 4 5 Participant Strongly Strongly disagree disagree neutral agree agree avg Q1 5 4 4 5 5 4.6 Q2 5 5 5 5 5 5 Q3* 5 5 4 5 5 4.8 Q4 4 5 3 4 5 4.2 Q5 5 4 4 4 5 4.4 Q6 5 5 5 5 5 5 Q7 5 5 4 4 5 4.6 Q8 5 5 3 4 4 4.2 Q9* 5 4 4 5 5 4.6 Q10 5 5 4 5 5 4.8 Q11 4 5 2 4 4 3.8 avg 4.5 4.5 3.75 4.5 4.833333 * values have been converted to other end of Likert scale due to negative phrasing of the survey question. Table A2.
• Do you have any ideas for improvements? The best suggestion was to include a notepad so one could manually write calculations and take notes. It was also recommended to include a calculator tool. In addition the protractor was not the easiest to read. b. Structured Interview The objective of the structured interview was to get the participants creative input.
the authors that this is due to the system being only a prototype. With a fully functional system we expect to have improved results. However, even an average of “agree” means the interface is heading in the right direction.
PACKET A For the experimenter, ensure that the SAT-M prototype is running on the computer at the top level screen. Then ask the participant the following demographic questions.
Task 1 When instructed to do so, please press the “home button” and then determine the bearing and distance to target #12. Screen: Vector 21b Answer: Bearing 060, Distance 6000 Time to Vector 21b screen:__________ Time to determine Bearing and Distance: __________ Number of navigational errors:__________ Was bearing correct?_____ Distance correct?______ Task 2 When instructed to do so, please press the “home button” and then determine what frequency the radio is currently set to.
Task 4 When instructed to do so, please press the “home button” and then determine the 6 digit grid of the point plotted on the map. Screen: Active Pen Answer: 845931 Time to Active Pen screen:__________ Time to determine grid: __________ Number of navigational errors:__________ Was the grid correct?_____ Once they participant has completed the tasks, inform them to complete the survey found on pages 3 and 4 of training packet.
PACKET B Welcome to the Supporting Arms Trainer - Mobile (SAT-M) usability analysis. During the next 15–30 minutes you will be asked to work with a prototype of the training simulator. The purpose of the SAT-M is to bring the simulation center to the Marine. We are looking to develop training software that will allow Marines to conduct immersive Call For Fire (CFF) training on a mobile device. You will work with a prototype of the interface. None of the major functionality has been implemented yet.
Task 2 When instructed to do so, please press the “home button” and then determine what frequency the radio is currently set to. Frequency ____________ Task 3 When instructed to do so, please press the “home button” and then determine what your present position is in grid. Location ___________________ Task 4 When instructed to do so, please press the “home button” and then determine the 6 digit grid of the point plotted on the map. Grid ___________________ You have completed the last task. Thank you.
1. The overall interface is intuitive. 7 tier strongly disagree disagree neutral agree strongly agree 3. It was difficult navigating through the device to find the appropriate information while completing the tasks. strongly disagree disagree neutral agree strongly agree 4. A fully implemented system would provide high quality partial task training for a JFO. strongly disagree disagree neutral agree strongly agree 6.
12. Does the device accurately represent the real world physical motion required to conduct the task. Training with this device on a regular basis will improve my ability to conduct CFF in the field. 12. What did you like the most about the interface? 13. What did you like the least about the interface? 14. Do you have any ideas for improvements? Thank you for participating in the usability evaluation of Joint Forward Observer Training Suite—Mobile.
APPENDIX B. C. EXPERIMENTAL DOCUMENTATION RESEARCHERS GUIDE 1. Chronological Task Listing Recruitment—(To be completed one week prior to execution of experiment). The researchers will begin recruitment and selection process. Emails will be distributed soliciting participation. Flyers will be disseminated throughout the NPS campus. When potential participants contact the researchers, they will be informally pre-screened for experience in CFF training.
familiarize themselves with the interface. All participants will be allowed this opportunity regardless of experience level with software. (Task duration: 5 minutes, location: NPS, MOVES Lab) Scenario reset—The researcher will reinitialize the scenario for the participant. On the tablet running SAT-M the researcher will tap the reset button. On the laptop running Observer Simulator the researcher will navigate to the file menu and select reset scenario.
SAT-M (Tablet device)—Using a finger the participant will tap the lensatic compass icon, and then rotate the tablet device until the “technical vehicle” is acquired. Observer Simulator (Laptop PC)—Using the mouse the participant will navigate to the lensatic compass icon, click the icon and then using the mouse to rotate the view, locate the “technical vehicle”. Researcher will note the elapsed time to complete task, and count any navigational errors made by participant during task execution.
Observer Simulator (Laptop PC)—Using the mouse the participant will navigate to the icon used to generate and send the 6-line CFF, the participant will click the icon and the researcher will observe that they are complete. Researcher will note the elapsed time to complete task, and count any navigational errors made by participant during task execution. The researcher will reinitialize the scenario for the participant. On the tablet running SAT-M the researcher will tap the reset button.
this information back to the firing agency by tapping the “checkmark” box. The firing agency responds when shots are fired, and he participant acknowledges this by tapping the “shot out” icon. After rounds impact, the participant ends the mission by tapping the “end of mission” icon. This concludes the protocol.
Protocol “B”—In this protocol participants will repeat the previous list of tasks in protocol “A”, but the device will be swapped for the one that was not previously used (i.e. in protocol “A” if a tablet was used, then the participant will use the laptop in protocol “B”).
D. RESEARCHERS PACKET Virtual environment training experiment (Researcher) READ FIRST If the participant has no knowledge of CFFprovide correct answers to the questions below. SUBJECT Number _____ Call for fire knowledge: Please answer the following questions, which are typically known by a Joint Forward Observer. (1) How many mils are in a circle? ____6400_____ (2) Name two Methods of target location.
Protocol “A” READ FIRST (RESEARCHER) Researchers guide, prior to having the participant begin the protocol using the participant worksheet allow them three minutes of interface familiarization (freeplay). There is no time limit for Protocol “A”. Researcher’s guide: After the participant has completed the Virtual environment training experiment sheet and is ready to execute Protocol “A” make sure they are seated in front of the DVTE or standing in front of the bench with the Tablet, as appropriate.
Task #4—Locate and activate the icon used for transmitting the CFF brief. Start time:_______ Finish time: _______ Researcher will note the elapsed time to complete task, and count any navigational errors made by participant during task execution. The researcher will reinitialize the scenario for the participant. On the tablet running SAT-M the researcher will tap the reset button. On the laptop running Observer Simulator the researcher will navigate to the file menu and select reset scenario.
Navigational Errors: _______________ Were rounds ‘on target?’ Yes / No Protocol “A” SUBJECT Number _____ Inform the participant that protocol “A” is complete and have them complete the questionnaire for protocol “A.” While they are completing the questionnaire please note anything specific challenges that the participant had with the system or anything unusual or interesting that the participant did while executing the tasks below.
Protocol “B” READ FIRST (RESEARCHER) Researchers guide, prior to having the participant begin the protocol using the participant worksheet allow them three minutes of interface familiarization (freeplay). There is no time limit for Protocol “B”. Researcher’s guide: After the participant has completed the Protocol “A” qualitative survey and is ready to execute Protocol “B” make sure they are seated in front of the DVTE of standing in front of the bench with the Tablet, as appropriate.
Task #4—Locate and activate the icon used for transmitting the CFF brief. Start time:_______ Finish time: _______ Researcher will note the elapsed time to complete task, and count any navigational errors made by participant during task execution. The researcher will reinitialize the scenario for the participant. On the tablet running SAT-M the researcher will tap the reset button. On the laptop running Observer Simulator the researcher will navigate to the file menu and select reset scenario.
Protocol “B” SUBJECT Number _____ Inform the participant that protocol “B” is complete and have them complete the questionnaire for protocol “B.” While they are completing the questionnaire please note anything specific challenges that the participant had with the system or anything unusual or interesting that the participant did while executing the tasks below. Once the participant has completed the questionnaire for protocol “B” provide them with the Post-experiment Demographic Questionnaire.
E. PARTICIPANT PACKET Naval Postgraduate School Consent to Participate in Research Introduction. You are invited to participate in a research study entitled Virtual Environment Training on Mobile Devices, Supporting Arms Trainer-Mobile. United States Marine Corps 2012 Science and Technology Plan identifies a critical Training and Education gap in T&E STO-6: Warrior Simulation: “Marines need to train as they would fight as small units, particularly for dismounted operations.
Final questionnaire and debrief (five minutes) − Participants will be video recorded to ensure accurate data collection. − We expect a minimum of 32 participants in the research, and anticipate as many as 64. − All subjects will be exposed to the same experimental conditions. Location. The interview/survey/experiment will take place at the MOVES Institute, Naval Postgraduate School in the laboratory. Cost. There is no cost to participate in this research study. Voluntary Nature of the Study.
Statement of Consent. I have read the information provided above. I have been given the opportunity to ask questions and all the questions have been answered to my satisfaction. I have been provided a copy of this form for my records and I agree to participate in this study. I understand that by agreeing to participate in this research and signing this form, I do not waive any of my legal rights.
Protocol “A” READ FIRST (PARTICIPANT) SUBJECT Number _____ The following experiment is confidential. Nothing you do or answer will be related back to you in any manner. Thank you for your participation. There is no time limit. Please spend the next three minutes getting familiar with the device and software. The proctor will inform you when three minutes has expired. Basic CFF tasks: Task #1—Determine your current location using GPS and record the location.
Execute CFF brief: You will now generate and execute a CFF, the target is the tank you located in task #3. Task #5—Execute CFF. First: Transmit a POSREP (Position Report) to the FDC (Fire Direction Center) using self-location. Once the POSREP has been transmitted you are ready to create and transmit the three transmissions for the CFF.
READ FIRST The following experiment and questionnaire are completely confidential. Nothing you do or answer will be related back to you in any manner. Thank you for your participation. Please answer all of the questions below and hand to the proctor when you reach "STOP HERE.” You may ask the proctor questions at any time. There is no time limit. SUBJECT Number _____ Protocol “A” qualitative questionnaire: (a “4” means no strong opinion) 1.
8. The 3D view interface was intuitive. strongly disagree 1 2 3 4 5 6 7 strongly agree 9. The device accurately represents the real world physical motion required to conduct the task. strongly disagree 1 2 3 4 5 6 7 strongly agree 10. The overall interface is intuitive. strongly disagree 1 2 3 4 5 6 7 11.
Protocol “B” SUBJECT Number _____ READ FIRST (PARTICIPANT) What device did you use during protocol “A”? Laptop / Tablet Please spend the next three minutes getting familiar with the device and software. The proctor will inform you when three minutes has expired. Basic CFF tasks: Task #1—Determine your current location using GPS and record the location. Your current location: ________________________________.
Execute CFF brief: You will now generate and execute a CFF, the target is the tank you located in task #3. Task #5—Execute CFF. First: Transmit a POSREP (Position Report) to the FDC (Fire Direction Center) using self-location. Once the POSREP has been transmitted you are ready to create and transmit the three transmissions for the CFF.
READ FIRST The following experiment and questionnaire are completely confidential. Nothing you do or answer will be related back to you in any manner. Thank you for your participation. Please answer all of the questions below and hand to the proctor when you reach "STOP HERE.” You may ask the proctor questions at any time. There is no time limit. SUBJECT Number _____ PART III: Protocol “B” qualitative questionnaire: (a “4” means no strong opinion) 1.
8. The 3D view interface was intuitive. strongly disagree 1 2 3 4 5 6 7 strongly agree 9. The device accurately represents the real world physical motion required to conduct the task. strongly disagree 1 2 3 4 5 6 7 strongly agree 10. The overall interface is intuitive. strongly disagree 1 2 3 4 The questionnaire continues on the next page.
Protocol “B” qualitative questionnaire continued Circle one: 11. Which device was more intuitive to use: Laptop / Tablet 12. If the software on both devices were about equivalent I would prefer to use: Laptop / Tablet 13. If each device had the same feature set I would prefer to use: Laptop / Tablet 14. This device is more convenient to train with: Laptop / Tablet 15.
READ FIRST The following experiment and questionnaire are completely confidential. Nothing you do or answer will be related back to you in any manner. Thank you for your participation. Please answer all of the questions below and hand to the proctor when you reach "STOP HERE.” You may ask the proctor questions at any time. There is no time limit. SUBJECT Number _____ PART IV: Post-experiment Demographic Questions: 1. What is your primary military specialty? (Provide name of specialty) ________________ 2.
a. What was the name(s) of the virtual environment(s)? a. _____________________ b. _____________________ c. _____________________ 11. When you were at your most proficient with CFF, how would you rate that proficiency? Untrained Novice Average Advanced Expert 12. Given many duties of a forward observer are perishable, how would you rate your current proficiency in call-for-fire? Untrained Novice Average Advanced Expert 13.
THIS PAGE INTENTIONALLY LEFT BLANK 124
LIST OF REFERENCES American Forces Press Service. (2013). Secretary details results of sequestration uncertainty. Retrieved August 23, 2013, from http://www.defense.gov/news/newsarticle.aspx?id=119421 Apple Inc. (2013). iOS developer library. Retrieved September 15, 2013, from https://developer.apple.com/library/ios/documentation/general/conceptual/ devpedia-cocoacore/MVC.html Associated Press. (2012). Number of iPads sold by apple by quarter. Retrieved July 22, 2013, from http://finance.yahoo.
Headquarters Department of the Army. (1976). operator, organizational, and direct support maintenance manual (including repair parts and special tools list) field artillery trainer kits (with field artillery trainer M31) (technical manual no. 9-6920-361-13&P). Washington DC: Department of the Army. Headquarters Department of the Army. (1991). Field Manual (Fm) 6-30, Tactics, techniques, and procedures for observed fire. Washington, DC: Department of the Army. Kroemer, J. (2006).
United States Army Field Artillery School. (1989). Field artillery training devices, software and special texts. Field Artillery. (August) Walker, D. (2013). Trends in U.S. military spending. New York, New York: Council on Foreign Relations.
THIS PAGE INTENTIONALLY LEFT BLANK 128
INITIAL DISTRIBUTION LIST 1. Defense Technical Information Center Ft. Belvoir, Virginia 2.
