Effects of Praise and Reinforcements on Engagement -- A Study of Two Interfaces on the Same Activity


By Warren Buckleitner

Buckleitner, Warren (2006). The Relationship Between Software Design and Children's Engagement. Early Education & Development, 17 (3), 489-505.

This is a summary of my doctoral dissertation, which was completed in 2004. You can download the complete version here. The study supports the idea that designers and evaluators of interactive media products for children should pay careful attention to the degree to which the implementation of control mechanisms such as reinforcements can have substantial effects on children's interaction with the software.

RESEARCH QUESTION: Are there observable differences in child behaviors in two versions of the same software sorting activity, one with a high level of instruction and reinforcement (high computer control), the other with relatively few instructions and reinforcements (high child control)?

Introduction

There is an established body of research that has examined the interaction style between adults and children. Some studies measured behavioral outcomes, such as various aspects of the educational effectiveness of the interaction. In the "wait-time" study, Mary Budd Rowe (1974) observed that the average time teachers waited between asking a question and taking further action to elicit a response is about one second. When a student responds to the question, teachers wait, on the average, less than one second before reacting to the response. Rowe called these two time periods-- the period between asking the question and acting further, and the period between the student's response and the teacher's reaction-- wait time. By asking teachers to increase their wait time to between three and five seconds, she observed a 300% increase in the length of students' explanations (Rowe, 1974).

Teacher/child interactions have been documented in intrinsic motivation literature (see Ames, 1990; Brophy, 1981; Lepper, 1985; Smilanski, 1968; Stipek, 1988 to name a few). Directly related to the study described in this dissertation is the literature that considers the quality and quantity of a child's engagement with a given task, as influenced by an adult/child interaction style. This relationship has been documented by Gerald Mahoney and James MacDonald (2003) with a population of young children with and/or at-risk for developmental problems. When children and parents or caregivers participated in two types of interactions (didactic and responsive), a positive relationship was identified between a responsive interaction style and children's social and linguistic development (Mahoney & MacDonald, 2003; Wolock, 1990; McWilliam et al., 2003).

This study examined these types of relationships in an interactive media context. A computer classification activity was created that was modified to simulate two contrasting teaching styles, similar to the Mahoney & MacDonald technique. The first style, called "high computer control" (HICOMP) attempted to simulate a teaching style where the teacher carefully introduced each problem, and provided frequent praise and encouragement throughout the experience. As a result, the child had less control over the flow of events, making the experience less responsive. The second style, called "high child control" (HICHILD) presented the identical sorting experience with the instructions, praise and encouragement turned off. As a result, a child experienced more control over the events, resulting in a more responsive overall experience.

Control, which was varied by changing the quantity of instructions and reinforcements, served as the independent variable in the study. The dependent variable, the engagement of the child, was measured by counting observable child behaviors that were recorded on videotape. These included 1) the number of tasks completed, 2) the number of clicks, or attempts to influence the instruction flow, and 3) the length of time the child chose to spend with each condition.

The study population was 38 preschool-aged children. In addition to discrete continuous measures, anecdotes during data collection provided additional insights. What children said and how they behaved throughout the experience was also documented, specifically children's competency with the input device (a mouse), the ways in which they responded to the software interface, and any other observations that could possibly be related to their engagement with the activity.

The Results in Brief

The outcome variables were the number of tasks attempted, tasks correct, time with the activity, mouse clicks and a child rating of the experience. In addition, anecdotal observations documented child reactions to both settings.

Children in the high child control treatment were more active, completing more tasks (mean = 64 vs. 20; p < .05), clicking the mouse more times (mean = 129 vs. 73; p < .05), and getting more tasks correct (mean = 41 vs. 16; p < .05). Children rated both experiences highly, and spent about the same amount of time with each condition.

In the high computer control setting, there were more clicks per task (mean = 4.07 vs. 2.09; p < .05), and children had a higher accuracy level (mean = 85% vs. 68% respectively). In addition, ANOVA procedures suggested that younger choose to stay with the HICOMP experience longer than the older group of children.

This study helps connect the established principles of human/child interaction to computer/child interaction, including the role of external reinforcements and the level of responsivity of the interaction. The results of this study suggest that designers and evaluators of interactive media products for children should pay careful attention to the degree to which the implementation of control mechanisms such as reinforcements can have substantial effects on children's interaction with the software.

A Closer Look

The measures revealed some interesting, statistically significant (p < .05) relationships. Children in the high child control setting performed more mouse clicks (129 vs. 73) and had lower accuracy rate for problems (68% vs. 85%), in about the same amount of time.

However, the most striking finding was that children attempted over three times more problems (64 vs. 20) and more than twice as many correct answers (41 vs. 16) in the high child control condition. While no significant differences were found by gender or session administration, the age of the children did matter in terms of the amount of time spent with the task. The population's large developmental span (40 to 60 months) led to a high degree of variability in children's performance across the tasks. In addition to the expected differences due to cognitive abilities, some children seemed determined to test the outer limits of the activity.Considering that outliers were not removed from the data set and the population of young children was heterogeneous, especially with regard to developmental level, the resulting standard deviations for time, clicks and tasks were large.

Major Findings

1. Children clicked more in the HICHILD setting, but had fewer wasted clicks than in the HICOMP setting. For the purposes of this study, a click is defined as the two part motion (and up and down stroke) when children choose to interact with the software interface. The click was easily counted due to the distinctive sound associated with stroke, as well as the visual clues provided by screen events.
Mouse clicks were a useful behavioral indicator of a child’s motives, interests, ability level and activity level. This meaning of the click depended on the situation. The click could mean something either intended, or non-intended. Intended clicks might have a meaning such as "OK, I'm ready, lets get started", "I'd like to choose that cookie", "Is this cookie the right one?" or an expression of affect such as "hey, hurry along" or "I'm feeling very frustrated!"
In the high child control setting, children clicked more (mean = 129.08 vs. 73.68 respectively; p < .05) over the same amount of time as the high computer control setting. This outcome has more meaning when interpreted in the context of the number of problems completed in each setting. In the HICHILD setting, children attempted more than three times (320%) the number of tasks (63.8 vs. 20.4; p < .05), resulting in a click per task ratio nearly two times (194%) that of the HICOMP setting (4.07 vs. 2.095; p < .05).
clicks.png
To conclude, when responsivity was increased, children were much more active, clicking more frequently; and more of those clicks were related in some way to an intended outcome (from the perspective of the software designer). In the HICOMP treatment, the added narration and reinforcement statements seemed to create a barrier to child's activity and problem solving effort.

2. Children chose to spend about the same amount of time in each trial. But there were interesting interactions for time and age. One initial hypothesis of this study was that the lower the level of responsivity, the less time a child would voluntarily stay with treatment.
Contrary to this prediction, children in the HICOMP setting actually spent a bit more time than when in the HICHILD condition (mean = 480 seconds vs. 540 seconds, respectively) although this relationship was not significant (p > .05). The novelty of the sorting experience kept the children in both situations, and children's willingness to comply with the software's instructions might explain the slightly greater time spent in the HICOMP situation. Because both treatments were conducted out of a child’s regular play setting in a parent lounge down the hall, it is possible that the novelty effect was magnified. This could be further explored with an additional study that would examine child reactions to each treatment in the context of a free choice period, over the course of a longer period of time, along with a hidden counter in the software to measure the frequency of use. In this study, these questions were not formally addressed.

The ANOVA revealed some notable findings when the entire group of children was divided by younger and older age groups. The 14 younger children, aged < 50 months on average chose to stay with the experience longer than the 22 older children (p < .05) regardless of the experimental condition.
An explanation for this may be the challenge level, which started with three objects to sort, based on one attribute, and increased to five objects and three attributes. Because most of the problems were geared toward the middle of the age group (46 to 52 months), the older, more competent children more quickly exhausted the novelty and challenge available in the experience than the younger group, resulting in a loss of interest, and less time on task. For designers, this helps illustrate the importance of having a fluid challenge level that either automatically adapts to the child's ability level, or that lets the child have some control over the challenge setting.

3. Children attempted more problems and experimented more in the HICHILD setting. Another statistically significant relationship was the number of problems that were attempted between the two treatments. In the HICHILD condition, the children were 317% busier, attempting 63 problems in approximately the same amount of time spent in the HICOMP condition with only 20 problems solved (p < .05).
When children experienced a more structured and controlled interface with a high level of narration and direction, they showed a decrease in activity, as measured by number of problems attempted. Anecdotal observations supported this observation, with more fidgeting, yawning, and placing head on the table during the HICHILD situation.
tasks.png
Another observation relevant to this topic was that the HICOMP treatment work was more accurate, with a higher percentage of correct answers (84.95% vs. 67.97% respectively; p < .05). When there was increased activity, there was a decrease in accuracy. When the sum of correct answers, however, was compared between the two conditions, during the HICHILD condition children ended up with 393% more correct answers -- 41.0 vs. 16.1 (p < .05).
Interpreting the significance of this finding is dependent upon the theoretical framework and associated instructional objectives of the software designer. If the end goal is for the learner to solve a higher number of correct answers and increase the amount of experimentation, the HICHILD setting is the preferable design. If higher accuracy regardless of the number of problems is the only goal, the HICOMP setting is the preferable option.

4. The older group of children chose to spend less time in the HICHILD setting than the younger group of children. When the population was grouped into two parts by age (over 50 months and under 50 months), there was a significant and interesting difference in the amount of time the two groups choose to stay with the activity. Regardless of the experimental condition, the younger group stayed longer than the older children (p < .05) although the HICHILD setting held them longer. (610 vs. 442 seconds, whereas the HICOMP setting was 573 vs. 567 seconds).
It may be that the increased responsivity of the HICHILD setting allowed the older, more competent children to use up the challenge sooner, resulting in a shorter session time. The greater number of attempted problems by the older children (80.57 vs. 52.10) supports this observation, particularly when compared with the older children in the HICOMP setting, where the difference narrowed to 20.6 vs. 20.2. The decreased responsivity seemed to be associated with the child’s ability to quickly reach their challenge level, which had an affect on the amount of time they chose to stay with the activity.

5. Children liked the game. They rated both experiences highly, but anecdotal observations seemed to indicate that children generally preferred the HICHILD treatment over the HICOMP treatment. A formal measure of the child's feelings about each treatment was attempted using a Likert-type scale. There were no significant differences between the two groups (4.65 for HICHILD, 4.58 for HICOMP; p > .05). When children were asked "how did you like it?" immediately after a treatment, they would say either nothing or that they liked it, by touching one of the smiles faces. It was hypothesized that children would rate the HICOMP experience lower than the HICHILD setting. This was not supported by the survey ratings.

Additional information was gathered less systematically, by observing children's reactions when their turn came up to play the second trial. In general, they would respond enthusiastically to the idea of coming back to the room to play the game some more, regardless of the first condition they experienced; high or high computer control.
Nine of the children, generally older, were able to verbally compare the HICHILD and HICOMP treatments after the second session. From these videotaped conversations, it was possible to determine that these children had more positive things to say about the HICHILD experience. In order to more accurately understand children's reactions to each treatment, additional exposures to both the HICHILD and HICOMP treatments would be necessary, over a longer period of time. It is likely that children would have a more discriminating attitude toward between the two treatments after the novelty of the experience is reduced.

Classification of Mouse Clicks Listed by Frequency During the Cookie Critters Activity

This is an attempt to classify the types of mouse clicks, or mands, observed in this study.

1. Double Stroke, Intentional Clicks. This click consisted of one complete down and up stroke while on the intended target. For example, the child sees a cookie, moves the cursor to it, and clicks. This type of click was more common in the older group of children (>50 months) who were more likely to have prior mouse experience. This type of click was common in both HICOMP and HICHILD settings.

2. Single Stroke, Intentional Clicks. Approximately 1 in 5 children used "drag and drop" or "hold and go" (Strommen) single stroke clicks in both the HICHILD and HICOMP settings, even though the activity used a "sticky mouse" making this technique unnecessary. A child using this type of click would first position the cursor over the target cookie, and then make one downstroke, holding down the mouse button, and not letting it come back up until it was over the target critter. This type of click requires the coordination of both fine motor and gross motor movements simultaneously. It was interesting that some children switched to this strategy in the HICOMP setting, from intentional clicks, after they learned that they could not speed the events along. Perhaps this was out of frustration.

3. "Hurry Up!" Unintentional Clicks. This click resulted when a child attempts to influence the temporal sequence of events on the screen by clicking the mouse. Commercial early childhood software activities that allow children to "click through" introductions or screen events may reinforce this behavior. This clicking behavior was observed only in the HICOMP setting.

4. "Rapid Fire" or "Machine Gun" Unintentional Clicks. This technique refers to when child sends a continuous stream of clicks, sometimes in a short burst and other times for longer sequences. The child's thinking seems to be along the line of "I’ll just keep clicking until the computer hears me." It was also a way to keep busy, perhaps creating a simulated feeling of control in the HICOMP setting. This was rare in HICHILD settings, much more common in the HICOMP setting when children did not have as much control.