Introduction

Defence Research and Development Canada (DRDC) provides science and technology advice to the Canadian Armed Forces (CAF) including the evaluation of embedded computer-based technologies to improve training effectiveness. This study aims to develop a proof-of-concept embedded Intelligent Tutoring System (ITS) that assists marksmanship skills training. An ITS is commonly defined as a computer-based system that provides customized and immediate feedback to trainees (e.g., Nwana,1990; Psotka et al., 1988; Sleeman & Brown, 1982). ITS incorporates some form of artificial intelligence (AI) and it requires minimal to no support from a human instructor. It usually includes four major components (Sottilare et al., 2014): (a) a domain model that contains expert knowledge and some typical errors and deficiencies that students typically experience; (b) a learner model that represents the current state of knowledge of the user, letting ITS know who it is teaching; (c) a tutor model that enables ITS to know how to teach by taking inputs from domain and learner models; and (d) a user interface that registers and converts student’s inputs from one form (texting, clicking, voice) to another (text, graphs, videos).

A training system is embedded when its components constitute an integral part of a device or piece of equipment. Witmer & Knerr (1996) defined Embedded Training (ET) as “a training concept … built into or added to a weapon system.” The two main benefits of embedded trainers are that they are (1) embedded into operational hardware so the training they provide is an accurate representation of their equipment, and (2) they are more accessible than formal, instructor-lead training, which increases the amount of training time students can realistically receive.

Depending on the structure and methods of AI, an ITS can be classified into four types: simple reflex, goal-based, model-based, and learning systems (Russell & Norvig, 2016). The simple reflex type includes the basic response systems that can react directly to the current environment. Such systems can be built using simple if-then rules and do not usually require the use of elaborate task representations or cognitive architectures. The other three types of models usually require cognitive architectures or task representations. Goal-based systems contain formal descriptions of their intended final states and selection mechanisms that reduce the discrepancy between their current states and their goal states. Learning system type ITS includes mechanisms that allow them to autonomously gather information and update their internal states. The ITS presented in this paper is a training system that combines elements of a simple reflex system with some more advanced mechanisms of a goal-based AI system. The following section provides a background in the field of motor-cognitive training systems, thus helping to place our ITS in the context of other similar systems, show its advantages and limitations, and discuss how our system can updated to become a learning system.

Intelligent Tutoring Systems for Motor-Cognitive Skills

Intelligent Tutoring Systems have been successfully used in multiple academic, industrial, and military settings (for reviews, see Kulik & Fletcher 2016; Nkambou et al., 2010; Nwana, 1990; VanLehn, 2011). Nevertheless, the majority were designed to train knowledge-based subjects—such as mathematics, economics, or computer science (e.g., Steenbergen-Hu & Cooper 2014). The use of an ITS to train cognitive-motor skills has remained limited up until last few years.

Cognitive-motor skills are hard to train and evaluate outside of a controlled environment, requiring portable training tools to record and track body movements and assess trainees’ motor performance in real time (Goldberg et al., 2018; Hodaie et al., 2018). However, recent advances in portable technologies, large data processing, and sensor development lead to the emergence of new ITSs in such diverse fields as medicine (Almiyad et al., 2017; Alvarez et al., 2015; Skinner et al., 2018), sport (Lee & Kim, 2010), driving (Ropelato et al., 2018), and industry (Hodaie et al., 2018; Marinescu-Muster et al., 2021; Westerfield et al., 2015). Despite the wide proliferation of these applications, limited amount of them was tested and reported in the research field of AI in education (Santos, 2016), the observation shared by another recent review of psychomotor intelligent tutoring systems for training motor-cognitive skills (Neagu et al., 2020).

In military, Mulgund and colleagues (1995) developed an Intelligent Flight Trainer (IFT) aimed at training pilots for helicopter flight proficiency. The tutor in this instance provided coaching and performance monitoring functions through a synthetic voice generator. The testing of the IFT in a training simulator showed that the system was successful in helping to improve skills related to maneuvering a helicopter. Rickel & Johnson (1999) investigated a virtual agent-based system that can be used as both a synthetic instructor and a virtual team member. Their training system, “STEVE” (Soar Training Expert for Virtual Environments), was designed for training procedural tasks related to mechanical maintenance and repair of naval ship equipment. The synthetic agents guide the trainees in a shared virtual environment, interacting through a voice processing module. STEVE proved to be a useful addition to naval operating procedures training despite its limited speech production and a somewhat narrow selection of tasks.

More directly related to the current study is work examining deployment and testing of ITSs to improve motor-cognitive marksmanship skills. Goldberg and colleagues (2018) developed an embedded, portable ITS using the Generalized Intelligent Framework for Tutoring (GIFT) architecture. The framework allows a quick authoring and developing of training applications (Sottilare et al., 2012). Goldberg and colleagues analyzed marksmanship performance in a shooting simulator, collecting data from multiple sensors embedded into an instrumented weapon (e.g., butt pressure, respiration rate, barrel movement, trigger squeeze pressure). While the system is still being validated experimentally (including user-interface testing), it is one of a few training systems that can assist with teaching a cognitive-motor skill.

Another ITS-based marksmanship tool was developed by Yeh & Ritter (2012). Their Declarative to Procedural Moving Target Tutor (D2P/MTT) helps trainees to learn to engage moving targets. D2P/MTT uses a step-by-step introduction to the basics of shooting moving targets, such as estimating the distance to the target, the speed of the target, and the angle of approach. Each of these elements is introduced individually to trainees, allowing to transform factual knowledge that later become part of their procedural knowledge. Later in training, these elements are combined, and the process repeats until the trainees achieve a target level of performance in adjusting a point of aim for distance and speed of the target. In experiments, the D2P/MTT training tool was tested with both college students and active-duty Marine instructors. The D2P/MTT-based training was compared with typical in-class instructions and showed an advantage in boosting participants’ procedural knowledge of adjusting a point of aim (Yeh & Ritter, 2012).

Even though the development of ITSs for training motor-cognitive skills is increasing, most of training systems described above were not tested experimentally. The tutor presented in this paper attempts to fill this gap by evaluating and comparing an ITS-based training with a standard training technique modelled after currently used by CAF marksmanship training.

The Moving Target ITS

The tutoring system presented in this paper helps shooters to improve marksmanship skills required to engage moving targets. The task includes several motor-cognitive skills and drills that are part of standard marksmanship training: estimating distance to the target, estimating the speed of the moving target, adjusting point of aim (POA) for a moving target, analyzing a missed shot, and learning tracking techniques. (for review, see Schendel & Johnston 1983). Below, we describe the MT-ITS, including the functional components and mechanisms, and how they interact in a training context.

Moving-Target ITS (MT-ITS) is a software-based tutor embedded into the digital sight of an instrumented rifle. The components of the tutor are shown in Fig.1. The system is positioned into Virtual Immersive Soldier Simulator, a test bed that provides the virtual setting for the tutor (VISS; Fig.1, top panel). The tutor includes an instrumented rifle with a digital sight and a top-mounted tracking system (Fig.1, bottom-left panel), and the Smart Sight System that assists trainee to properly aim moving targets (Fig.1, bottom-right panel). The VISS environment allows a tracking system to pinpoint the orientation of the rifle and the shot location. When a shot is fired at a moving target, the target continues to move during the time of flight of the bullet. Thus, it is necessary to aim in front of the target, otherwise shots will fall behind it. This aiming in front, to anticipate the movement of the target, is known as lead. The amount of lead depends on the speed and distance to the moving target; the Smart Sight System is responsible for processing and delivering this information to trainees.

Fig. 1
figure 1

Components of MT-ITS: VISS (top image), instrumented rifle (bottom left), and Smart Sight (bottom right). (Note: The bottom-right panel shows Smart Sight System aiming reticle with full target telemetry (range, speed, Military Grid Reference System location), shown as a vertical arrow and a calculated point-of-aim indicator to hit the target’s center of mass shown as a red box.)

Full size image

MT-ITS’s two feedback mechanisms—a real-time correction of aiming (delivered by the Smart Sight system) and an after-action feedback—comprise the main software components of MT-ITS, representing the “intelligence” of the system. The former guides trainees to adjust their target-tracking behaviour in real time and the latter provides a customized feedback after each target engagement. Both mechanisms were developed using Application Scripting Interface of Virtual Battle Space 2 (VBS2) simulation package (bisimulations.com). The VBS2 scripting language is capable of a wide range of functions available through hundreds of script commands, which satisfy the requirements of the feedback that MT-ITS provides.

For the real-time correction, the Smart Sight System calculates the optimal point of aim and presents it to a trainee in a form of red rectangle superimposed on the target, as shown on the bottom-right panel of Fig.1. The small red square shows the qued aiming location adjusted for speed and distance to the target. MT-ITS presents the same type of moving target (generated as a combination of distances, directions of movement, and speed) with and without the aiming point sequentially, repeating this combination of qued and unqued presentation for all types of targets. The process assumes that multiple instances of targets and corresponding aiming points are transferred into memory, serving to improve and automate trainee’s performance. This training mechanism is based on both theoretical grounds and practical considerations. The method of repeated presentation of multiple cued instances is based on the Instance Theory of Automatization (Logan, 1988), which assumes that practice leads to storing of multiple instances of trained material in memory, thus increasing the speed of retrieval and contributing to effortless performance. On a practical side, MT-ITS incorporated features that were added on advice and suggestion of experienced combat arms soldiers of the CAF. These soldiers were recruited to provide feedback and advice regarding the design of MT-ITS and its consistency with CAF marksmanship training requirements. The feedback prompted a comprehensive design modification addressing nearly all aspects the MT-ITS (e.g., target parameters, settings of virtual environment, feedback form, and many others).

For the after-action feedback, MT-ITS records a trainee’s performance for each target engagement, including hits, misses, position of shots, shot spread (grouping), and the locations of the points of impact. This after-action feedback is delivered in two ways: (a) a sequential presentation of each shots fired (Fig.2, left panel) and (b) a statistical summary of all shots fired in the same trial (Fig.2, right panel). The literature shows that blocked feedback (summarized by 5, 10, or 15 trials) improves the retention of skills in transfer studies (Schmidt et al., 1989). Moreover, presenting feedback on only the immediately preceding trial might be detrimental to training effectiveness (Lavery & Sudden, 1962); one possible explanation is that immediate feedback might overwhelm trainees with information and foster heavy reliance and dependence on this form of feedback. For the same reason, the cued trials were combined with non-cued ones.

Fig. 2
figure 2

A training session feedback screens in MT-ITS training condition. The left panel shows a feedback screen generated after each shot and the right panel shows a summary of all shots fired at the same target

Full size image

The feedback that MT-ITS provides are of two types: Knowledge of Results (KR), which provides information about the outcomes of the action or the movement, and Knowledge of Performance (KP), which provides information about the nature of the action or movement (for further details see Schmidt et al., 2018). Both types of augmented feedback are important for learning a motor skill effectively, but most studies up until last twenty years employed KR due to difficulties of capturing and analyzing body motions (Santos, 2016). While MT-ITS was envisioned as a training tool incorporating both types of feedback, its current implementation principally presents KR feedback in the form of shot statistics. KP feedback is partially supplied through the Smart Sight, where trainees adjust the aiming point. The possibility of enhancing KP feedback in MT-ITS is reviewed in the Future Works section.

In terms of standard ITS component described in Introduction, MT-ITS incorporates aspects of the domain and tutor models of ITS, as well as a user interface. The learner model is largely undeveloped in this iteration of the system.

The MT-ITS’s domain module includes some key metrics of optimal marksmanship performance, as considered by expert marksmanship instructors: group sizeFootnote 1 (the distance between the two most distant shot locations), distance between the mean point of impact and the target’s centre of mass, and hit rate. These metrics are compared with observed shooting performance and then incorporated into after-action feedback.

The learner model requires details of each trainees’ knowledge states and, as such, it is not well-elaborated in MT-ITS. We limited the applicability of our training tool to a novice shooter. Consequently, the selection of the training settings was tuned for this specific group. With a future plan to include a detailed evaluation of trainees’ marksmanship level, MT-ITS can be modified to customize its training difficulty, depending on the trainee’s skill and knowledge level.

The tutor model takes input from the domain and the learner’s models in order to customize its feedback to trainees. MT-ITS incorporates this information in both forms of feedback it gives to trainees: as a guide to adjust point-of-aim in real time and as a detailed performance feedback in after-action review. In its previous version, MT-ITS provided some tips and corrections as an after-action feedback (Zotov et al., 2017). The content of tips was based on the discrepancy between observed performance and expected optimal performance. For example, if most of the shots were lagging the target, the tip would be “Your aim is generally lagging behind the ideal aimpoint – increase lead significantly”. The tips were not included in the present version due to their limited training value (the issue is addressed in the General Discussion section).

The goal of the user interface is to convert the formats of trainee’s inputs from one form to another. MT-ITS records trainee performance as data in VBS2 script format and converts it to texts, images, and diagrams presented to trainees in the two forms of feedback outlined above.

Overall, MT-ITS aims to improve a standard training Small Arms Training (SAT) simulation ranges used by Canadian Armed Forces for marksmanship training. The differences between two training settings are in using Smart Sight System and the amount of feedback that participants receive. What follows is our empirical efforts to compare training that includes MT-ITS with training that does not. The objective was to test these two training methods using the same virtual environment and the same amount of training.

Experiment

Two training methods were tested in the experiment: an ITS-enabled training that used MT-ITS tool and a Standard training that replicated a typical training that soldiers of Canadian Armed Forces would receive in a SAT setting. The differences in these two training conditions were the types of feedback. For Standard training, trainees received no feedback during target engagement and limited feedback after engagement, which consisted of showing target sheet with points of impact marked. For the ITS-enabled training, trainees received aiming feedback in the form of the Smart Sight and a detailed after-action feedback, described above. The experimental design includes two control sessions (pre-training and post-training) and a training session between them, thus the difference between the methods can be tested in the analysis of pre- and post-training sessions. The analysis of the training session adds some details of the learning dynamics.

Method

The experiment included three sessions: (1) a pre-experimental session (reading and signing information forms, engaging in a warm-up exercise), (2) an experimental session consisting of pre-training (shooting different types of moving targets), training, and post-training blocks (similar to pre-training) and (3) a post-experimental session, in which participants filled out training evaluation questionnaires. Table1 shows the duration of each part; more detail will be provided below. Experienced combat arms soldiers of the Canadian Armed Forces were recruited to provide feedback and advice regarding the design and features of ITS-enabled training and Standard training conditions.

Table 1 Sequence and duration of experimental blocks
Full size table

Participants Twenty-eight participants were recruited for this experiment. All participants were from Reserve Canadian Army units. The participants were split equally into two groups corresponding to the two training conditions (Standard and MT-ITS) with fourteen participants in each group. The participants had little to no experience with firearms and had served less than two years in the Canadian Armed ForcesFootnote 2. Defence Research and Development Canada (DRDC) Human Research Ethics Board reviewed and approved the study protocol.

Apparatus and Stimuli The study was conducted in the VISS – a DRDC-built research testbed used to examine marksmanship performance and small arms-related technologies. The simulation is based on Bohemia Interactive Simulations’ Virtual BattleSpace 2 (VBS2) software, with custom modifications that allow a fully motion-tracked mock weapon to aim at and engage virtual targets in a CAVE-like background environment (Fig.1, top panel) spanning roughly 150 degrees of the shooter’s field of view, projected across three 3m screens. An OLED micro-display fitted into a Trijicon Advanced Combat Optical Gunsight housing acted as a simulated weapon sight and was mounted on a modified airsoft rifle made by WETech, patterned after a short-barrelled 10.3” M4 variant. Though the mock rifle did not physically recoil when rounds were fired, virtual software recoil was added to the simulation to prevent participants from rapidly firing multiple rounds without having to re-acquire the target. Reflective markers on the weapon allowed it to be tracked by a NaturalPoint OptiTrack infrared camera system. The shooter was centred in the simulator, roughly three meters away from the background screens, in a seated-supported position to reduce discomfort and fatigue-induced aiming instability over the course of long trial sessions.

The virtual environment consisted of an outdoor field scene with a long wall perpendicular to the shooter located either 100 or 250m away, with two gaps cut into it (Fig.3). A human target appeared in one of the two gaps and attempted to run across it at one of two speeds. The participant’s task was to hit the target as quickly and as accurately as possible before it reached the end of the gap. To equalize exposure time between slower-moving and faster-moving targets, gaps were 22 and 40m wide. The arrangement of the gaps and the distance to the walls was randomized and counterbalanced across trials.

Fig. 3
figure 3

A three-walls, two gaps layout for the pre- and post-training sessions

Full size image

Depending on the session and condition, two types of moving targets were used: an animated target (Fig.2) and a simulated NATO E-Type silhouette target, 1m tall, 50cm wide (Fig.4). The surface areas of the silhouette of the E-type target and the animated character were approximately equal.

Fig. 4
figure 4

A training session hit feedback screen in the Standard condition, with a NATO E-Type silhouette target

Full size image

Procedure During the pre-experimental session, participants read information materials, signed consent forms, watched training videos, and engaged in a shooting warming-up exercise (Table1). The printed information materials explained the procedure and provided some basic principles for engaging moving targets. The training videos demonstrated the principles of shooting moving targets and presented information of how to adjust POA depending on the speed and the distance to target. For the MT-ITS condition, the video contained additional information on how to use the Smart Sight feature of the MT-ITS. The videos were designed and developed specifically for this study using selected military marksmanship training materials and advice from an Advanced Small Arms and Urban Operations qualified instructor. The Advanced Small Arms course is the Canadian Armed Forces infantry qualification prerequisite for the rank of Sergeant. Its focus is to standardize and evaluate the instructional and coaching methods used by candidates when teaching the use of small arms to new and experienced soldiers. The Urban Operations Instructor course is an advanced course that focuses on training soldiers of all experience levels to successfully engage the enemy at close ranges.

After completing the pre-experimental session, participants continued with the experimental session, consisting of warm-up, pre-training, training, and post-training blocks (Table1). Warm-up trials served to familiarize participants with the VISS, the airsoft rifle, and the task by engaging 20 moving targets of varying speeds and distances. It was followed by the pre-training block, which was conducted to establish baseline shooting performance for each participant. After a short break, the pre-training block was followed by the training block, in which participants were engaged in a condition-specific coaching. After a break, participants started the post-training block, which had a similar content to the pre-training block and served to test the effectiveness of training. Prior to each block, the instrumented rifle was digitally zeroed (calibrated), ensuring that the weapon sights aligned with the tracking system.

At the end of the experimental session, participants answered a questionnaire about the quality of the training tools. The questions solicited information related to the utility and training benefits of the training tools, asking participants’ opinions about the structure of the training program, the number of targets presented, and their opinions about the usefulness of the information presented in the training. Participants were also asked whether they felt they learned something new, and whether the training methods could become a useful addition to marksmanship training. The responses were rated on a five-point scale starting from “Strongly Disagree” (corresponding to 1 point) to “Strongly Agree” (5 points). The scale of the “Number of Targets” question varied on a three-point scale (“Too few targets” - “About right amount” - “Too many targets”, corresponding to -1, 0, 1 rates).

Sessions and Conditions The pre- and post-training sessions consisted of 96 trials each. On each trial, one of eight possible combinations of three target parameters was presented: (1) two distances (100 and 250m), (2) two speeds (8km/h and 15km/h, about the speed of a jog, and a fast run, respectively), and (3) two directions of movement (left-to-right and right-to-left) in all possible triplets. Pre- and post-training sessions were partitioned into four 24 trials chunks, with a short, participant-controlled break between them (Table1). For each trial, participants were given up to eight shots to successfully engage the target. In pre- and post-training sessions, after a target was hit, it dropped, and a new trial was initiated. The pre- and post-training sessions were identical and did not differ between two conditions.

There was a rest break halfway through the session. For each part of the training session, participant engaged 64 targets. These targets were obtained in the same way as in the pre- and post-training sessions by combining all possible combinations of distance, speed, and direction of movement. The training session used a pre-set sequence of target presentations: participants in both conditions started with easy-to-hit targets (100m, slow, either direction), continued with more challenging targets (100m fast, 250m slow, either direction), concluding with the hardest to hit targets (250m, fast, either direction). Considering that participants’ performance for easy-to-hit targets was close to ceiling in the previous experiment (Zotov et al., 2017), the proportion of these targets was lower: Table2 shows how many trials for each target type were given to participants during the training session. The moving target’s exposure time was approximately 12s. As in pre- and post-training session, participant could use up to eight shots.

Table 2 Number of training trials as a function of speed and distance
Full size table

The two training conditions differed on the type of instructions that participants received: a standard training (Standard) and an ITS-based training (MT-ITS) with partial KP feedback. The conditions differed in terms of the shape of target, the type of feedback, and whether a point-of-aim cue was presented. In the Standard training condition, the participants engaged an E-type silhouette target, moving either left-to-right or right-to-left between two barriers. Table3 summarizes the differences in Standard and MT-ITS conditions. On each trial, participants could use up to eight shots to hit the target. Even when a target was hit, it would not drop to allow participants to engage as many shots per targets as they could. After a target completed its movement, participants received visual feedback in the form of a target diagram with the location of all hits marked on it (Fig.2). The visual feedback was presented on an auxiliary monitor (Fig.1, top-left corner) and in the digital sight; participants could use either one depending on their preference.

Table 3 Training sessions details of MT-ITS and Standard conditions
Full size table

For the MT-ITS training condition, participants were shown an animated target moving across the range. Presentation of each target type was repeated, alternating between showing and withholding a target cue. On the cued trial, participants were shown the correct POA in the form of a small red square joined to the centre of mass of the target by a red line (Fig.2). As in the Standard condition, participants could use up to eight shots to hit the target. Whenever a hit was registered, the heads-up display flashed a red bracket around the reticle to provide additional real-time feedback. After a target completed its movement (each target’s exposure was approximately 12s), participants received visual feedback in the form of sequential presentation of a diagram for each shot taken in a trial with the point of impact and the optimal POA of the target marked on it.

Like in the Standard condition, the visual feedback was presented on both an auxiliary computer monitor and on the digital aiming sight. Participants were able to advance the feedback slides at their own pace using a button on the mock rifle. After reviewing each shot, participants were shown a summary slide that showed the location of all hits and near misses for that trial. The MT-ITS training session consisted of two parts that differed on the focus of the feedback: the first block emphasized accuracy by indicating hit percentages, while the second block emphasized engagement speed by indicating “time-to-kill”, which was the timing lapsed between the onset of the target’s movement and the moment when the target was hit. To encourage participants to be more efficient and engage the target faster in this second block, the target would drop after it was hit three times on the same trial.

Results

The data analyzed in this section are from the pre-training, training, and post-training sessions. The dependent measures were accuracy of shooting and lead distance. Accuracy of shooting is the hit rate for each target type calculated as the ratio of hits for each target, multiplied by 100. The number of shots to hit a target could vary (up to eight shots per target), but this measure was not reported, considering that its value depended on the different shooting styles of participants: some preferred to use as few shots as possible and some would use multiple shots. The lead distance (referred to hereafter as “lead”) is the horizontal difference between the observed and the optimal POA in cm; with lead lagging behind optimal POA coded negative and lead ahead of the optimal POA coded positive. The lead analysis tested if there were any systematic biases in tracking moving targets. Considering that the recorded lead is an instantaneous value at the moment of shot release, it does not provide any tracking behaviour over the course of the entire engagement.

The independent variables were training condition, distance to target, speed of target movement, and training (Pre- versus Post-training). The results were analyzed separately for testing sessions (comparing performance before and after training) and for a training session. The final part of this section explores the results of the post-experimental questionnaires.

Pre- and Post-Training Sessions

The results of pre- and post-training sessions were analyzed in a mixed factor ANOVA with three within-subject factors (distance × 2, speed × 2, training × 2) and one between-subject factor (training condition × 2). We used an alpha level of 0.05 for all statistical tests. The direction of target movement has no effect on participants’ performance, therefore the left-to-right and the right-to-left data was collapsed. Table4 presents the results of the ANOVA for both accuracy of shooting and lead measures. Considering a small sample size, the ANOVA was tested for homogeneity using Levene’s test for equality of variances, which returned no significant value for all tested pairs.

Table 4 Effects of distance and speed of targets as function of training and condition
Full size table

There was a strong effect of training (for accuracy, F(1, 26) = 48.41, p < .001), indicating that in the post-training session participants’ performance had improved. There was also a significant interaction of training with condition, F(1, 26) = 4.99, p < .05, indicating that the performance gain in MT-ITS training condition—for both shooting accuracy boost and a reduction in aiming lead—was significantly stronger than in the Standard condition. That is, the MT-ITS training had produced significantly better shooting accuracy and shorter discrepancy between optimal and observed lead distances than the standard training. The significant interaction of Condition x Distance x Training shows that MT-ITS was especially effective for distant targets (250m). The effect of Condition was not significant, confirming that performance in two training groups differed only in post-training sessions. The effect of training was significant on its own and in combination with a distance (Training x Distance and Training).

Table5 shows a summary of performance for average hit rates and lead as a function of distance, speed, training, and condition. Shooting accuracy was inversely related to the distance to the target and the speed of the target; that is closer and slower targets were hit with higher accuracy. Shooting accuracy for the easiest targets (100m, slow) was at the ceiling, approaching 100%. As expected, the hardest targets to hit were fast-moving distant targets (250m, fast): even with eight rounds at participants’ disposal, the targets were hit only around 60%. The effect of distance and speed on performance were significant, confirming that it was harder to hit distant, faster moving targets. What also worth mentioning is that the training was especially effective for distant targets; in both training conditions the average gain in hit rates was around 15% (unlike 3–4% for easy-to-hit 100m targets).

Table 5 Mean values of performance measures as a function of speed, distance, and training
Full size table

The lead measure fell into a bimodal pattern: for 100m targets participants were leading too far ahead (leading too much), but for 250m targets participants were not leading enough (lagging behind). One possible explanation to this finding is that simultaneous change in distance and target speed result in a non-linear change in the lead distance, leading to the underestimation of the lead—a well-documented human bias of estimation of non-linear trends (e.g., Kwantes & Neal 2006).

Training Trials

The objective of the analysis was to test whether performance increments during training trials follow a growth function and to analyze the shape of the function. While the evidence of a training effect can be revealed through an increase in a linear, power or exponential function (Anderson, 1982; Heathcote et al., 2000), the linear function is the simplest function to test the observed data fit increasing, decreasing or a flat trend (Pedhazur & Kerlinger, 1982). If there is a positive practice effect, then the training data would fit an increasing function.

The linear function takes two parameters: b, which is a slope of the function, and a, which is an intercept. The slope indicates whether the best fitting line follows increasing (positive value) or decreasing (negative value) function. The value of intercept parameter is not important for analysis of training trials as it simply a translation parameter that places the curve on the Y-axis without changing the shape of the linear function.

The trial-to-trial data from the training trials were pulled for each type of target and then fit to a linear function. Recall that participants in the MT-ITS condition were trained with cued trials (presenting the location of the optimal POA for each target). These cued targets were easier to hit; as a result, the hit rate for these trials was higher. Thus, these trials were excluded from the sequence analysis to avoid contamination of the training performance by these boosted trials. The analysis included 128 training trials per participant for the Standard condition and 64 training trials per participant for the MT-ITS condition. The number of the same type target types is presented in Table2.

To avoid obscuring individual training trends by averaging, each participant’s training data for each type of target (100m, 250m, separately for fast and slow targets) was fitted to a linear functionFootnote 3. using SPSS’ CURVEFIT method, which reports significance of the fit and a and b parameters of the fit. The slope values of each participants’ fit to training data (b parameter) are shown in Table6. The positive values indicate an increasing function (or a positive training trend) and the negative values indicate decreasing function (or a negative training trend). As the table shows, individual results are quite diverse: there are cases of increasing, flat, and decreasing trends. Nevertheless, most of the training curves in the MT-ITS condition show significant increasing function while most of the curves in the Standard condition show flat curves. The training trials in the MT-ITS condition showed a consistent increasing function for all types of targets; participant performance was improving from trial to trial. The training trials in the Standard condition showed a less consistent pattern; the growing function was evident for some participants. Figure5 shows the collapsed curves (averaged for each condition) for training trials in both conditions. To show the training curves on the same scale, Fig.5 shows average gains in hit rates, thus the scales’ range (-10 to 10) reflects a relative gain in performance expressed as percentage. The figure demonstrates the overall trend in practice: the training trials in the MT-ITS condition (top row) show a gradual improvement of shooting accuracy with practice, whereas the training trials in the Standard condition (bottom row) do not show a similar trend.

Table 6 Slope values of linear function fits to participants’ training data
Full size table
Fig. 5
figure 5

Hit rates for training trials for both conditions. (Notes: For MT-ITS Condition, only non-que trials are presented to avoid contamination by the higher accuracy cued trials)

Full size image

Post-Experimental Questionnaires

The post-experimental questionnaires asked participants about the quality of the training tools: the structure of the training program, the number of targets presented, the usefulness of the information presented in training, the novelty of the training method, and the potential of the training methods become a useful addition to marksmanship training. Table7 presents the average values of the participants’ ratings for each group. Overall, the participants in the MT-ITS condition gave more favourable ratings than those in the Standard condition, but the differences were not significant.

Table 7 Mean values of participants ratings of the training tools
Full size table

Summary

The results of the experiment demonstrated that the MT-ITS training surpassed Standard training, helping to improve shooting accuracy and increased the precision of aiming at the moving targets. The analysis of training trials revealed that that training rates with MT-ITS were more consistent in terms of producing steady gains in performance with training; the same trend was not observed in the Standard condition. While the effect size corresponding to the advantage of MT-ITS training over the Standard CAF training was small, it was impressive considering that the training lasted less than an hour, our sample size was relatively small, and the participants rate of improving was very diverse.

The advantages achieved by the MT-ITS marksmanship training against the “holes in paper” approach used in the Standard condition might reflect the ability of the MT-ITS to overcome the limitations inherent in the existing live-range procedures. Unlike the conventional training, MT-ITS supplies trainees with a detailed information on their performance. For example, it supplies trainees with the locations of the missed shots, which give an opportunity for participants to adjust their aiming points for speed and distance. It shows each engaged target sequentially, allowing trainees to link certain techniques or strategy to good or poor performance and make the necessary adjustments. Finally, it helps to adjust point of aim with the help of cue delivered by the Smart Sight system, thus helping trainees to perceiving intuitively the correct point of aim. The advantage of Smart Sight system was especially obvious during training: the majority of our participants in MT-ITS condition steadily progressed, unlike CAF participants, whose progress was not as steady. Possibly, in the absence of cued trials in the Standard condition, participants used a trial-and-error approach in estimating the proper POA.

Other results were consistent with a typical performance at marksmanship training: all participants performed well shooting slow targets at close range (100m). Performance was noticeably worse for shooting fast targets at longer ranges where accuracy dropped significantly. The study was consistent with a trend often observed by marksmanship instructors: the accuracy of shooting was inversely related to the distance and speed of the moving target. The direction of the target movement had no effect on performance.

General Discussion

MT-ITS Modifications

The version of MT-ITS presented in this paper includes modifications that helped it to overcome some of the shortcomings of the previous version (Zotov et al., 2017). That version included text-based tips that informed participants of any systematic errors they were making while engaging moving targets. The feedback in the previous version also included detailed statistics, such as the standard deviation of all shots and the coordinates of each point of impact. While it sounds like a useful feature, participants in the previous study stopped paying attention to this feedback after only a few trials. In the absence of behavioral and physiological responses collected (e.g., trigger squeeze pressure, barrel movement, respiration), the content of these tips became repetitive and redundant; some participants reported that the detail of the provided statistics was overwhelming.

Tips enhanced by rich sensory information can be re-introduced in the next iteration of MT-ITS (discussed in the Future Work section), but the most beneficial form might not include verbal tips. Considering that at the cognitive/associative stage, (which presumably trainees reach during the training sessions), the learners benefit the most from brief, non-verbal corrections, preferably presented in graphical forms (Singer et al., 1994). The sequential presentation of each engagement including observed and optimal POA for each shot, was added to help trainees pinpoint each shot they fired in terms of aiming accuracy.

The experimental design underwent several changes too. In the previous study, only one training session was used, whereas this version included two blocks. The first part was focused on accuracy, while the second added a speed component, thus mirroring cognitive and associative stages of skill acquisition theory (Fitts & Posner, 1967). The bulk of recruited participants were novices identified by their short duration of service (up to two years) and little to no experience shooting moving targets. This was done to maximize the benefits of training. Participants were provided with the post-experimental questionnaires to find out how useful the training tools were.

MT-ITS Limitations

Despite its benefits, MT-ITS does have several limitations related to poverty of sensor inputs, restrictions of the simulation environment, and deficiencies of instrumentation, placing the system in a category of an advanced feedback training system with some AI elements rather than a fully-fledged ITS. A coaching system, such as MT-ITS, would benefit from adding a number of sensors to enhance quality of feedback, as the work of Goldberg et al., (2018) showed (e.g., sensors for barrel movement, trigger squeeze, recoil, and respiration).

The experimental sessions were conducted in the VISS, a synthetic environment lacking certain features of the real world. One obvious way to improve on the “realism” of the VISS is to update both the instruments used in the simulation and the software. The experimental weapon used in the study was an airsoft rifle; although it shares many similarities with the real weapon upon which it is based (an M4 CQBR), some characteristics, such as recoil and especially trigger weight, pull, break and reset are notably different. A more realistic weapon would bring the simulated training closer to live training. A better ballistic modelFootnote 4 than the one used in the simulation would improve fidelity and accuracy of firing. The resolution of the simulation can be further enhanced to allow higher, more accurate representation of the shooting environment. Nevertheless, it is worth mentioning that even the most “immersive “setting may not yield any increase in training effectiveness and may create a less effective training environment, as Smallman and colleagues (2007) noted.

Unlike a true embedded training system, which is installed in the actual operation hardware (in this case, a soldier’s issued weapon and sight system), the VISS is a large external simulator which cannot be deployed with the trainee or provide a 100% accurate representation of the hardware’s features and handling characteristics. While this study examined the content that might be featured in an embedded training system, the benefits of physically embedded systems were not tested. In particular, the greatly increased access and training time afforded by being installed in their own weapon, ready to use at any time, was a factor that was actively avoided to maintain an even comparison against the standard training content.

The MT-ITS condition used the same animated target types for all sessions, whereas the Standard condition mixed animated targets in pre- and post-training sessions, and “E-type” targets in the training session. The main reason for using “E-type” targets in the Standard condition was to replicate the typical marksmanship training as closely as possible, but this change added an artifact of switching types of targets. Possibly, this change has no effect on performance: in our 2017 study, a standard condition with “E-type” used as training targets (and animated targets for pre- and post-training sessions) still showed significant advantage against a control, no-training condition (Zotov et al., 2017), but in order to confirm that the process of switching targets had not affected the performance in the Standard condition, the same target type should be used in both conditions. Further, it is not clear what feature of MT-ITS was responsible for training effectiveness. Was it the Smart Sight, the enhanced feedback, or some other factors (or their combination) that contributed the most to the training gains? This study can only confirm that MT-ITS has been a more effective training system than a typical training that soldiers are trained with, but more research would be required to reveal individual contributions of each of the MT-ITS training features. By adding two new experimental conditions, the effect of individual contributions of the Smart Sight system and the enhanced feedback (and their possible interaction) can be identified and measured.

Considering that the participants were tested right after the training, it is important to investigate the retention of the skill learned by ITS-based training. Re-testing participants in several intervals (e.g., after 1 month and 6 months) is necessary to find out whether their initial training gain persists over time and by how much. In the previous study (Zotov et al., 2017), six participants were re-tested six months after they completed their first experiment. The performance level of the re-tested participants was on par with performance in their post-training session six months earlier and the difference between pre-training and re-test sessions was approaching significance. Nevertheless, with only six returning participants, the sample size was very small, and the analysis lacked enough statistical power to detect significant differences on the order of those observed; a larger sample size to obtain a definite answer is required. To better evaluate which of the training systems participants valued more, the two training tools need to be provided to the same participants so they can compare them directly, so that the post-experimental questionnaires can reflect which of two training conditions participants prefer and why. Finally, the effectiveness of MT-ITS must be tested in a transfer of training trial, where two groups of participants would be trained with either MT-ITS or live ranges and then tested in a live fire exercise. We hope that the MT-ITS training would be as effective as the live range training, considering that other marksmanship simulators showed their effectiveness in the transfer of training studies (e.g., Grant 2013; Jensen & Woodson, 2012; Yates, 2004).

Future Work

As we pointed out, the main weakness of MT-ITS lays in its “low intelligence”. This deficiency imposes considerable limits on its ability to provide a feedback customized for individual trainee’s needs. To improve training effectiveness of MT-ITS, we propose several enhancements related to the type of feedback it provides and to the system interface mechanism it uses. These enhancements are consistent with those that Santos (2016) outlined and suggested for any future motor-cognitive ITS, such as adding sensing movement and designing and delivering new forms of feedback.

To enhance the quality of real-time feedback we propose addition of a number of sensors to monitor: a butt pressure, a barrel movement, a trigger squeeze, and a heart rate and respiration monitors. A butt pressure sensor measures the force with which a rifle’s butt is pressed against a trainee’s shoulder, optimally constant and within a specific range. Too much or not enough pressure can negatively affect tracking behaviour. A barrel movements sensor detects excessive barrel movements, which can correlate with excessive butt pressure or ineffective breathing. A trigger squeeze sensor detects amount of finger pressure on a trigger, possible revealing unproductive trigger pressure during a target engagement. A respiration monitor can detect some inefficient breathing prior to a target engagement (e.g., not holding a breath). Taken together, information from these sensors can be incorporated into a real-time feedback and guide trainees to adjust their aiming behaviour. The feedback format can be visual (as a diagram showing optimal and observed values of the sensor readings), auditory (as a pitch that changes depending on the readings along with some verbal feedback), and haptic (as vibrations triggered by too much butt pressure or excessive barrel movements). This form of KP feedback might greatly enhance the quality of information that a trainee gets in a real time. Finally, a hear rate monitor can supply trainees with a way to estimate their stress level (detected by to elevated heart rate and decreased heart rate variability, for more information see Thayer, 2012). Integrated feedback could improve these techniques with such techniques as Tactical Breathing (Bouchard et al., 2012).

The after-action feedback can be enhanced by applying a tracking algorithm to analyze performance on a number of previous trials. This memory system would receive input from the sensors that we described above, allowing some ineffective patterns of engagement to be identified, shared with the trainees, and corrected. These ineffective patters can be detected by comparing observed and optimal shooting behaviours, but the implementation of such mechanism requires advanced expert model. Goldberg et al., (2018) presented a case of such a model for marksmanship training, which was obtained by collecting performance and behaviour metrics of shooting experts and then inputting these data into a regression analysis to obtain the weight values for each metric to evaluate shooting behaviour. The form of feedback could be a combination of text, visualization, and synthetic speech. Considering the amount of customized information such a feedback would include, it might overcome the problem of being too repetitive and redundant, as we observed in the verbal feedback that we included in our early versions of MT-ITS.

The system interface could be customized depending on skills and knowledge level of a given trainee assessed during baseline evaluation and training sessions. As we noticed, the baseline shooting performance of some of our participants was high and the gains in hit rates were relatively low (and even were reduced by the end of training session). With ability to customize training session, MT-ITS would adjust the difficulty levels, including detecting plateaus in performance for some target types. The system would sample performance data from a number of previous trials and calculate slopes for different types of targets; as the slope values flatten, the training algorithm will stop presenting this type of target. Another customization enhancement can be achieved by a performance tracking method that would allow to quickly evaluate a trainees’ progress and to identify the most effective training intervention. One of the ways to implement such method was demonstrated in Fenza, Orciuoli, and Sampson study (2017), which presented a concept of ITS that uses a reinforcement learning with artificial neural network to observe a trainee’s actions at different states of the learning process. The selection mechanism of the tutor relies on a Next Task Selection algorithm that evaluates a trainee’s performance and selects a new task to keep a trainee in her zone of proximal development, which is an optimal level of task difficulties most conducive to training improvements.

The effects of psychological stressors have been extensively investigated in recent years (for review, see Vartanian, Boscarino, Jarmasz, & Zotov,2022). With added ability to measure the level of stress, the future testing environment of MT-ITS might include a number of stressors to bring the training condition closer to a live training or even to a combat setting. The stress level during marksmanship training can be induced by presenting multiple targets simultaneously, adding weapon malfunctions, including hostile fire, and placing trainees in a competitive environment. Monitoring stress level will allow to place a trainee in the optimal task difficulty zone to achieve the highest training benefits (Guadagnoli & Lee, 2002). Overall, the proposed enhancements that we listed above will substantially increase the autonomy and the level of customization of MT-ITS, placing it in a category of learning AI systems

Conclusion

Despite its limitations, MT-ITS is one of the first AI-based systems for coaching motor-cognitive skills that has undergone extended testing against other training methods. The GIFT-based marksmanship training system (Goldberg et al., 2018) is another example of AI-based system that has undergone extensive testing against both simple instruction and human instructor and might provide a good comparative system to evaluate training effectiveness (personal interaction with the first author). Marksmanship training with MTT/DP2 was extensively tested, but the MT/DP2 tutor aims to teach only the cognitive aspect of the marksmanship skill (specifically, how to adjust for target characteristics). Unlike MTT/DP2, MT-ITS helps master actual skill rather than part of it.

MT-ITS may eventually be able to overcome known limitation of human instruction in the teaching of discrete cognitive-motor skills such as marksmanship. Rauter and colleagues (Rauter et al., 2011) characterized these as inability to hold attention and concentration over a longer period of time, no access to important physiological and biomechanical variables that characterize movements, and considerable limitations in providing augmented concurrent feedback. As any other motor-cognitive ITS, our tutoring system would be vigilant through the entire training session, have a great potential to access numerous biomechanical and physiological variables, and could give a real-time feedback that includes multiple streams of collected information. The results of this study show that an embedded training system can provide substantial training augmentation —if not a partial substitution for live training—that might be efficient, economical, and flexible in its ability to create a wide range of scenarios. There are considerable potential benefits of delivering and practicing the skill through an ITS that is embedded in the training tools. An autonomous ET system that requires little to no human support might become a practical alternative to teach motor-cognitive skills, providing useful automated coaching in the absence of a qualified instructor.