Computer vs. Workbook Instruction in Second Language Acquisition

Noriko Nagata
University of San Francisco

Abstract:
Many studies have failed to demonstrate the effectiveness of CALI (Computer Assisted Language Instruction) as compared to non-CALI instruction, and the pedagogical significance of CALI programs has been questioned. This study employs an intelligent CALI program, called Nihongo-CALI (Japanese Computer Assisted Language Instruction), which uses natural language processing to provide more sophisticated error analysis and feedback than conventional CALI programs. The study compares the effectiveness of Nihongo-CALI with non-CALI workbook instruction, in the form of self-study lessons.
The results of the study show that given the same grammar notes and exercises, ongoing intelligent computer feedback is more effective than simple workbook answer sheets for developing learners' grammatical skill in producing Japanese particles and sentences. The results emphasize the importance of providing an intelligent level of feedback to the learner, a task for which natural language processing is well suited.

INTRODUCTION

Computers have long been regarded as a potential tool for accelerating the learning of a foreign language. Unfortunately, many attempts to incorporate computers into foreign language teaching have met with failure. Pederson (1987) reviews several comparative studies done in the past (late-1960s to mid-1980s); for example, CALI (Computer Assisted Language Instruction) versus language lab instruction (Morrison and Adams 1968), CALI versus traditional classroom learning (Adams and Rosenbaum 1969; Barrutia 1970; Williams 1980), and assignments on the computer versus written tasks with paper and

53

pencil (Mellgren 1983). In these studies no significant differences in learning were found. Kleinmann (1987) employed twenty different commercial software packages to compare the effectiveness of the CALI materials with non-CALI materials but found no significant difference between them in reading achievement. See also Roblyer, Castine, and King (1988) and Dunkel (1991) for reviews of the effectiveness research on computer assisted instruction. Pederson points out that the above studies attempted to attribute learning gains to the medium itself rather than to the way the medium was utilized and to the way the materials were designed. Kleinmann comments that most of the software programs used in his study function as little more than electronic textbooks. It is true that if CALI programs simply transfer printed textbooks to the computer screen, the computer is nothing more than an electronic page-turner, and we cannot expect better results from CALI. Many other features could be incorporated into CALI programs, however. For example, Teichert (1985) compares the effectiveness of CALI with conventional workbook exercises for learning German vocabulary and grammar. In his study, the CALI modules were able to provide immediate feedback, suggestions for review, repetition to a point of 80 percent mastery, and on-line "help," which the conventional workbook does not possess. The results of his study show that the CALI treatment group scored 10 percent better on the post-test than the workbook group. The study suggests that utilizing the full potential of CALI may increase instructional effects. A further problem is to determine which feature(s) actually have a positive effect (Kulik and Kulik 1987). One promising feature of CALI programs is their potential for providing detailed, relevant, and immediate feedback in response to student errors. The present study examines the effectiveness of such feedback in comparison to traditional workbook exercises.

Kulhavy and Wager (1993) state that in the past three decades there have been more than 50 studies comparing various feedback configurations in programs or program-like lessons, but there appears to be no consistent pattern of results. For example, some studies show that "complex" feedback (or "elaborative" or "explanatory" feedback) (i.e., feedback containing more information) produces higher instructional performance (e.g., Albertson l986; Collins, Carnine, and Gersten 1987; Grant, McAvoy, and Keenan 1982; Hannafin 1983; Roper 1977). Some studies show that increases in the amount of information provided have no significant effect on lesson performance (e.g., Corbett and Anderson 1990; Gilman 1969; Hodes 1985; Kulhavy, White, Top, Chan, and Adams 1985; Merrill 1987). Kulhavy and Wager conclude that "both the content of feedback and the timing of its presentation influence learner performance" (p. 15). They propose that one approach to developing effective feedback is to perform a detailed analysis of the types of errors made by students, and then to attempt to design a feedback strategy aimed at specific error correction (e.g., Anderson, Boyle, and Reiser 1985; Birenbaum and Tatsuoka 1987).

54

In the field of computer assisted language instruction, it has been pointed out that most of the existing CALI programs possess only limited capabilities for analyzing learners' responses, diagnosing their errors, and providing feedback (Garrett 1987; Lou, Wang, and Hung-Yeh 1992; Nagata 1993). Lou, Wang, and Hung-Yeh (1992) discuss the weakness of currently available EFL (English as a foreign language) software packages as follows: "many software packages require users to type in simply a letter or a number and answer judging in the software packages is usually unsatisfactory because they simply use naive string matching techniques" (p. 25). Liou, Wang, and Hung-Yeh compare the effectiveness of a computer courseware package with that of written homework assignments in the context of teaching English composition to native speakers of Chinese. Both the CALI courseware and the homework assignments contain ten lessons which consisted of grammar exercises. The homework sessions were simulations of students working on their assignments at home. The homework sessions were conducted in the classroom, and after the students finished the assignments, they picked up answer sheets to correct their answers by themselves. The result shows no significant difference between the CALI group and the homework group on the post-test. One reason for this outcome might be related to the kind of error analysis and feedback provided by the CALI program. The program employs standard keyword matching and error anticipation techniques capable of providing detailed feedback for some types of student errors. However, since it does not parse the student's response, the range of possible errors for which it provides detailed feedback is restricted. The study therefore invites the question whether a significant difference could be found in a similar study using a CALI program equipped with parsing technology that is capable of providing detailed feedback to a wide range of possible student errors.

Conrad (1996) notes that since the mid-1980s, CALI effectiveness research has shifted to the comparison between different types of CALI programs, instead of comparing CALI with traditional instruction. Hence, one now sees few empirical studies on the relative effectiveness of CALI vs. non-CALI. However, when it comes to utilizing computers for second language instruction, the question of whether and when computer programs can be more effective than traditional non-computer instruction is still a basic question to be addressed. Since the quality of computer software has been greatly improved using state-of-the-art computer technology, it may be possible to obtain better instructional effects from CALI than we could expect before. The purpose of this study is to employ an intelligent CALI system incorporating parsing technology, and to compare the effectiveness of intelligent CALI with traditional written workbook instruction. The intelligent computer program used in the study is called "Nihongo-CALI" (Japanese Computer Assisted Language Instruction). Nihongo-CALI employs an artificial intelligence technique, natural language processing, to parse the learner's input. Because it uses this technology, it can provide more sophisticated error analysis and feedback

55

than conventional CALI programs (Nagata 1992, 1993, 1995). The possibility of an intelligent level of computer feedback has been discussed for over ten years (e.g., Pusack 1983; Higgins and Johns 1984; Underwood 1987, 1989; Garrett 1987; Bonner 1987; Fischetti and Gisolfi 1990; Orey and Nelson 1993). Also, many researchers have been developing intelligent CALI systems using natural language processing (i.e., parsing techniques) and they describe how the systems work and what they can do (e.g., Si-Quing and Luomai 1990; Yazdani 1991; Labrie and Singh 1991; Levin, Evans and Gates 1991; Liou 1991; Sanders 1991; Loritz 1992; Holland et at. 1993; Juozulynas 1994; Hagen 1995). However, there have been very few empirical studies demonstrating instructional effectiveness of intelligent CALI in second language acquisition (Nagata 1995). Technical progress on the development of intelligent CALI systems is indeed vital for the future of CALI, but at the same time we need more empirical studies to discern the conditions under which intelligent CALI is more effective than other traditional (and perhaps less expensive) types of instruction.

The present study examines whether an intelligent CALI program (i.e., Nihongo-CALI) can obtain better results than a traditional written workbook in teaching a second language at the college level. There is no general way to compare Nihongo-CALI with workbook instruction because there are different ways to incorporate workbook instruction into language teaching. However, since Nihongo-CALI is designed to be used as supplementary work outside of the classroom, the present study compares Nihongo-CALI with workbook instruction, in the form of self-study lessons in which students practice the target language by themselves. Both provide the same grammatical instruction and exercises. The main difference is that Nihongo-CALI provides ongoing detailed grammatical feedback in response to the learner's errors, while workbook instruction provides answer sheets to have the students check their responses without any detailed feedback targeted to the individual errors. The feedback provided by Nihongo-CALI is elaborative and explanatory because it is based on detailed grammatical analyses of the student's attempts at producing Japanese sentences. The target structures are Japanese sentence constructions involving the Japanese particles, ga, o, wa, ni, and de. The following explains the Japanese particles and sentence constructions briefly.

PARTICLES AND SENTENCE CONSTRUCTIONS JAPANESE

Japanese uses particles to mark the grammatical and semantic roles of noun phrases. For example, the particle ga indicates the subject who performs the action, the particle o the object that the action operates upon, and the particle de the location where the action takes place. In the sentence Sumisu-san ga Nihon de susi o tabemasita¹ "Mr./Mrs. Smith ate sushi in Japan," Sumisu-san ga indicates the subject, Nihon de the location, and susi o the object. Although Japanese has a canonical SOV order, the word order before the verb is

56

relatively flexible, so the above sentence may be expressed as Nihon de Sumisu-san ga susi o tabemasita, Susi o Sumisu-san ga Nihon de tabemasita, and so forth. Table 1 presents a brief summary of particle functions used in the study.

Particles Functions Example Sentences 1. GA Subject who performs the action [Suzuki-san ga] kimasu yo. "[Mr./Ms. Suzuki] is coming." 2. GA Subject and object whose states [Kaataa-san ga] [Nihongo ga] yoku are described by a stative predicate. wakarimasu. "[Mr./Ms. Carter] understands [the Japanese language] well." 3. O Object that the action operates upon. [Omosiroi hon o] kaimasita. "I bought [an interesting book]." 4. O Location through which the action [Kooen o] arukimasita. moves. "I walked [in a park]." 5. NI Direction toward which the action Kinoo [gakkoo ni] ikimasen desita. moves or the goal of the action. "I didn't go [to school] yesterday." 6. NI Location where the non-activity [Gakkoo ni] atarasii konypuutaa ga referent is located. arimasu. "There are new computers [at school]." 7. DE Location where the activity takes [Gakkoo de] kaigi ga arimasita. place. "There was a meeting [at school]." 8. DE Instrument by means of which the [Waapuro de] tegami o kakimasita. action occurs. "I wrote a letter [with a word processor]." 9. WA topic and/or contrast of the sentence. [Sono keeki wa] Tanaka-san ga tukurimasita. "[As for that cake], Mr./Ms. Tanaka made it."

Table 1. Japanese particle functions used in the study

As seen in Table 1, the same particle can mark several different functions and different particles can mark similar functions that may not be distinguished in the first language (e.g., o, ni, and de indicate the location but there are fine semantic distinctions between them). Certain pragmatic functions such as topic or contrast (marked by wa ) may not occur in the first language. There is no clear, one-to-one correspondence between Japanese particles and English prepositions which serve a roughly similar purpose. Accordingly, students are easily confused by the Japanese particles. In fact, the Japanese particle system is a serious obstacle for second language learners of Japanese at the college level. This would be an area in which the aid of explicit grammatical instruction and ongoing detailed feedback on errors would be particularly useful (Nagata 1995, 1997). The Nihongo-CALI program used in this study provides such detailed grammatical feedback for learners' particle errors.

57

Using the Nihongo-CALI system, Nagata (1993, 1995, Nagata and Swisher 1995) previously studied the effectiveness of intelligent feedback and two types of traditional feedback for teaching Japanese passive and active sentences. Intelligent feedback explains grammatical and semantic functions of particles in response to the learners' particle errors, while traditional feedback does not. The first type of traditional feedback indicates only missing or unexpected particles and the second type also indicates the positions of missing particles, but no detailed grammatical explanations on the errors were provided. The results reveal that intelligent feedback is significantly more effective than both types of traditional feedback. As seen in these results, the effectiveness of CALI varies depending on what kind of feedback is provided to the students. This may explain why there have been many conflicting results from the CALI effectiveness research. Results may depend on whether the study employs intelligent CALI or traditional CALI. Traditional CALI is similar to self-study workbook instruction in the sense that it rarely explains the precise nature of individual learner's errors. On the other hand, intelligent CALI can provide detailed feedback targeted to specific deficiencies in the learner's performance. The present study compares the effectiveness of intelligent CALI (Nihongo-CALI) with workbook instruction, focusing on the difference in their abilities to provide feedback. The study investigates whether there is a significant difference between these two types of instruction in the acquisition of Japanese particles and sentence constructions.

NIHONGO-CALI AND WORKBOOK INSTRUCTION

Both Nihongo-CALI and workbook instruction include four lessons of exercises on Japanese particles and sentence constructions. Each lesson consists of grammar notes (i.e., a short text describing the target structures) and sentence-production exercises in which students are asked to type Japanese sentences in light of a provided communicative context.² The appendix includes the first page of the grammar notes provided by Nihongo-CALI. Figure 1 illustrates one of the Nihongo-CALI exercises as it is presented on the computer screen.³ Since those exercises are targeted for the students in First Semester Japanese, the Japanese writing systems (kana and kanji) are not incorporated into the exercises.

The students are asked to type a response sentence next to the arrow prompt on the bottom of the screen. By clicking "Options" on the left-top corner of the screen, the students obtain the option menu items such as "Grammar Notes" to see the grammar notes, "Vocabulary Hint" to see a list of words expected to be used for the exercise,⁴ "Next Exercise" to go to the next exercise, and "Previous Exercise" to go back the previous exercise. Both Nihongo-CALI and the workbook instruction provided in the study present exactly the same grammar notes, exercises, and vocabulary hints. The difference between Nihongo-CALI and workbook instruction is that Nihongo-CALI provides

58

Options What would you say in Japanese if you were in the following situation. In this exercise, do not drop any phrase particle. Your friend has asked if bus line 31 stops here. Tell her that no, that bus doesn't go on this road. >

Figure 1. A Nihongo-CALI exercise

ongoing feedback in response to the learners' grammatical errors, while workbook instruction is simply accompanied by answer sheets without any explanations on the individual errors.

Nihongo-CALI uses NLP (natural language processing) to analyze the learner's input sentence and to generate feedback messages. The NLP analyzer consists of a morphological parser (Hausser 1989), a syntactic parser (Tomita, et al. 1988), and data bases such as a core lexicon, a set of morphological rules, a set of syntactic rules, and feedback message templates (Nagata 1992). (A detailed description of Nihongo-CALI is provided in Nagata 1992 and 1995.) In response to the question in Figure 1, a student might type "Iie, sono basu ga kono miti ni ikimasu," which includes two particle errors and a verbal conjugation error. A correct answer would be something like "Iie, sono basu wa kono miti o ikimasen."⁵ The following feedback messages are provided in response to this incorrect sentence:

Read the following messages:

<Particle error>

• GA is not expected to be used here.

• NI is not expected to be used here.

• WA is missing.
• O is missing.
• In this context, SONO BASU was already under discussion and is the "topic or contrast" of the sentence. Use the particle WA to mark it.
• In your sentence, KONO MITI is the "direction" which the action moves toward, but it should be the "location" through which the action moves. Use the particle O to mark it.

  59

<Predicate error>

• The predicate you typed is in the affirmative form. Change it to negative.

The errors are classified into vocabulary errors (i.e., missing/unexpected words), particle errors (i.e., missing/unexpected particles and incorrect use of particles), and verbal predicate errors (i.e., missing/unexpected verbs and incorrect use of verbal forms such as perfective for imperfective and negative or affirmative). If the student fails to give a correct answer three times, the option menu "Correct Response" is provided to permit the student to see the correct answer. The workbook instruction provides the same exercises on paper and the students are asked to write down an answer for each exercise in the space provided. The grammar notes and the vocabulary hint are also available in the workbook as in Nihongo-CALI. After the students in the workgroup finish all exercises, an answer sheet is provided. The answer sheet lists the correct answers to each exercise. The workbook students are instructed to correct their responses with a red pen, based on the correct answers provided. It should be, however, noted that it is possible to include more information on the answer sheets such as explaining some anticipated errors and solutions. Whether such information would significantly improve the effectiveness of workbook instruction is an interesting question for a future study.

THE STUDY

Two first-semester Japanese classes at the University of San Francisco were involved in this study. First, the same mid-term exam was administered to both classes and their scores were used to make sure that there was no significant difference between the two classes in the level of achievement in the course prior to the experiment.⁶ A total of twenty-six students (thirteen students in each class) participated in this study. The two classes were randomly assigned to either a CALI group or a workbook group. The students in both groups received six experimental sessions over the course of two weeks. Although the study concerns the relative effectiveness of CALI and workbook instruction as competing modes of self-study (outside of the classroom), the study was performed in class to control for the amount of time spent by each group. About fifty-minutes classroom time was allocated for each experimental session. The same instructor conducted all experimental sessions.

The first session was carried in the regular classroom: the students in both groups were given the same brief lecture (about twenty minutes) on the Japanese particles and sentence constructions, and were then given a pre-test (about twenty minutes). The pre-test consisted of production tasks such as 10 blanks to be filled in with appropriate particles and 3 questions requiring the student to construct Japanese sentences using the particles.

60

A perfect score on the pre-test was 22.5.⁷ The pre-test was analyzed to determine the learners' performance level in producing the Japanese particles and sentences prior to the computer or workbook sessions.

From the second to the fifth sessions, the CALI group accessed Nihongo-CALI in the computer lab and the workbook group received workbook exercises in the classroom. In each computer session, the students spent about five minutes reading the grammar notes (which explained the grammatical and semantic functions of the Japanese particles and sentence constructions and provided example sentences employing those particles), and then moved to the exercises. Each workbook session followed the same procedure as the computer session: the students were instructed to read the grammar notes first and then to move to the exercises. As noted, when the students were doing the exercises, they always had access to both the grammar notes and the vocabulary hint for that lesson. The students in the CALI-group received ongoing feedback for their errors and corrected them based on the feedback, while the students in the workbook group corrected their errors using an answer sheet provided when the exercises were completed. Both groups were also informed that they could leave the class after they finished the assigned lesson. Through four computer/workbook sessions, the students studied the subject ga, the object o, and the topic/contrast wa in the first session, the double ga, the location ni, and the location de in the second session, the direction ni, the instrument de, and the location o in the third session, and reviewed all particles in the fourth session. Both the computer program and the workbook materials were available to the students only during these four sessions.⁸

A questionnaire was administered at the end of the last computer/workbook session. On the questionnaire, the students were asked to rate 23 items by a 5-point scale (1 strongly disagree, 2 disagree, 3 undecided, 4 agree, and 5 strongly agree) and to provide comments on the computer/workbook sessions. The questionnaire was used to investigate whether there is a significant difference in attitude toward CALI or workbook instruction.

The sixth session was conducted in the classroom, where both the CALL group and the workbook group received a post-test. The post-test included all questions used in the pre-test (10 filling-in-the-blank and 3 sentence-production questions). In addition, 10 more fill-in-the-blank and 8 more sentence-production questions were included in the post-test. The format and difficulty of the additional questions were the same as for those in the pre-test. Since the post-test asked the students to "produce" the target language, an additional test requiring the students to "comprehend" the target language was administered. The comprehension test asked the students to write English equivalents of Japanese

61

conversations written in roman characters (involving 13 sentences including 17 particles). The sentences used in the post-test and in the comprehension test were all similar to those provided by the computer/workbook exercises. A perfect score on the post-test was 71 and a perfect score on the comprehension test was 13.⁹ The post-test was analyzed to see whether there is a significant difference between the CALI group and the workbook group in their respective abilities to produce both Japanese particles and basic sentences. The comprehension test was analyzed to see whether there is a significant difference between the two groups in comprehending basic Japanese sentences including the target particles.

Two days after the experiment, the post-test and the comprehension test were corrected and (temporarily) returned to both groups. The instructor explained each answer on the tests, and then collected the tests again because the same test was to be used as a retention test later. During the three weeks after the experiment, the students received speaking practice (a total of one and half hours), in which the students were paired and asked each other who did what and in where, etc., based on provided pictures. All questions required the students to use the particles they had learned.

Twenty-three days after the experiment, a retention test was administered. The retention test was identical to the post-test. The second comprehension test was also administered immediately after the retention test. The second comprehension test included similar conversations to those given in the first comprehension test (it consisted of 13 sentences including 18 particles). A perfect score on the retention test was 71 and a perfect score on the second comprehension test was 13.¹⁰ Both tests were analyzed to examine the long-term effects of Nihongo-CALI and of the workbook instruction.

Three days after the retention test (i.e., 26 days after the experiment), an oral test was administered by the instructor to individual students (about 5 minutes for each student). The oral test was designed to have the students discuss the activities of four people. First, the students were asked to describe in Japanese where two imaginary people (Mr. Tanaka and Ms. Carter) went yesterday and what they did, based on the pictures provided (e.g., Tanaka-san wa Gooruden Geito Burizzi ni ikimasita "Mr. Tanaka went to Golden Gate Bridge," Marin Heddoranzu de haikingu o simasita "He went (did) hiking in the Marin Headlands," etc.). Next, the students were asked to inquire of the instructor where two other people (Mr. Smith and Ms. Yamada) went and what they did. The oral test involved the production of 8 sentences, and the students were required to use the particles they had learned to construct those sentences. Each student's performance was recorded and transcribed. The perfect score was 28.¹¹ The oral test was analyzed to see whether there is a significant difference between the two groups in the oral production of Japanese sentences.

62

As noted, Nagata's previous study (1995) used two versions of the Nihongo-CALI programs, one providing intelligent feedback and the other providing enhanced traditional feedback, in order to compare the effectiveness of these two types of feedback. The present study employed the intelligent version of the Nihongo-CALI program and used the same mid-term exam, the same pre-test (expect for one sentence-production question added), and the same post-test as in the previous study. The subjects in both studies were the students in the first-semester Japanese courses at the same university which were taught by the same instructor using the same materials. Therefore, the data from the present study were compared to the ones obtained in the previous study in order to investigate whether there is a significant difference between the traditional version of Nihongo-CALI and the workbook instruction.

THE RESULTS AND DISCUSSION

The scores on the mid-term exam shows that the CALI group and the workbook group had no significant difference in their respective levels of achievement in the course prior to the experiment (t = 1. 1 7).¹² The result of the pre-test scores also shows that the two groups had no significant difference in the production of Japanese particles and sentences before they began the computer or workbook sessions (t 0.93).¹³ Table 1 presents the means and standard deviations of the two groups for the post-test scores, and the result of the t-test. (All statistical analysis in this study employs two-tailed t-tests) The result shows that the difference between the two groups in their Post-test scores is statistically significant at the 0.02 level, favoring the CALI group.

	CALI group	Workbook group	T-test
	Mean SD	Mean SD	t	Sig of t
Post-test	55.2 10.7	44.6 9.4	2.69	0.02

Table 1. The results of the post-test (scores out of 77)

Next, the results of the present study are compared with those in the previous study. In the previous study, students in the traditional CALI group were merely informed of missing/unexpected words and of the location of missing particles, while the intelligent CALI group received detailed grammatical explanations on the nature of the errors as used in the present study. When comparing the test performance of the workbook group with that of the previous traditional CALL group, there is no significant difference between the two groups, either in the midterm scores (t = -0.004), in the pre-test scores (t = -1.08),¹⁴ or in the post-test scores (t = -0.64). On the other hand, although there is no

63

significant difference between the workbook group and the previous intelligent-CALI group either in the midterm scores (t = -0.16) and in the pre-test scores (t = -0.83), the difference between the two groups in the post-test scores is significant at the 0.01 level, favoring the previous intelligent-CALI group (t = -3.08).¹⁵ This is consistent with the results of the present study. It indicates that the relative effectiveness of CALI and non-CALI (i.e., workbook instruction) varies depending on whether CALI is accompanied by intelligent feedback or traditional feedback. The results emphasize the importance of providing an intelligent level of feedback to the learner, a task for which natural language processing is well suited.

About three weeks after the experiment, the students took the retention test. The result of the retention test (Table 2) shows that the difference between the two groups in the retention test scores is statistically significant at the 0.05 level, favoring the CALI group. The result indicates that given the same follow-up instruction (i.e., feedback for the post-test and speal6ng practice), Nihongo-CALI is more effective than the workbook instruction in the long term.

	CALI group	Workbook group	T-test
	Mean SD	Mean SD	t	Sig of t
Retention test	56.0 8.2	47.1 11.1	2.33	0.05

Table 2. The result of the retention test (scores out of 71)

Besides the post-test and the retention test (which were both production tests), the students were given two comprehension tests: the first one was administered immediately after the post-test and the second one was given immediately after the retention test. The scores on both comprehension tests did not show any significant difference between the CALI-group and the workbook group (see Table 3). In the comprehension tests, the students were asked to interpret the written Japanese conversations described above. For example, the conversation involved a situation in which two people are asking each other about what they did yesterday (e.g., Kinokuniya de omosiroi hon o kaimasita "I bought an interesting book in Kinokuniya," etc.) It appears that once the students understood the meaning of each word in the sentence such as Kinokuniya (the name of the book store omosiroi hon ("interesting book"), and kaimasita ("bought"), they could interpret the meaning of the sentence easily without paying attention to the functions of the particles attached to the noun phrases.¹⁶ Flynn (1986) states that grammatical competence is less critical in comprehension than in production since other extralinguistic knowledge and information available to the subject can be used to make a coherent interpretation of the stimulus sentence. The results of the study confirm that sentence production involves

64

more syntactic processing than sentence comprehension (see also Swain 1985; Pica, Holliday, Lewis, and Morgenthaler 1989; Swain and Lapkin 1995; DeKeyzer and Sokalski 1997; Nagata 1996 for discussion on the role of output in second language acquisition). The results also suggest that a comprehension test alone may not measure the learner's grammatical skill accurately.

	CALI Group	Workgroup	T-test
	Mean SD	Mean SD	t	Sig of t
Comprehension test 1	12.2 1.0	11.8 1.5	0.92	NS
Comprehension test 2	12.2 0.9	11.6 2.1	0.91	NS

Table 3. The results of the first and second comprehension tests

(both scores out of 13)

Three days after the retention test, the oral test was administered. The oral test involved the production of Japanese sentences using the target particles. The result of the oral test (Table 4) shows that the difference between the two groups in the test scores is statistically significant at the 0.001 level, favoring the CALI group.¹⁷ As noted above, the students received oral practice after the experimental sessions. The result indicates that given the same follow-up speaking practice, Nihongo-CALI is more effective than the workbook instruction for the oral conversation tasks involved in the study. Since the students in the CALI group developed more grammatical skill than those in the workbook group through the instruction they received, this difference may be reflected in their oral performance.

	CALI group	Workbook group	T-test
	Mean SD	Mean SD	t Sig of t
Oral test	22.2 3.3	15.8 4.2	3.99 0.001

Table 4. The result of the oral test (scores out of 28)

Table 5 presents the means and standard deviations of the ratings for each item on the questionnaire (1 strongly disagree, 2 disagree, 3 undecided, 4 agree, and 5 strongly agree). Items 14 through 22 were given only to the CALI group because they were not applicable to the workbook group. The results of the questionnaire show that the students in the CALI group had very positive attitudes toward Nihongo-CALI, whereas the students

65

in the workbook group did not show such positive attitudes toward the workbook instruction. Although both groups received the same grammar notes, the CALI group rated item 1 ("The grammar notes are helpful,") and item 2 ("The grammar notes are written clearly,") significantly higher than did the workbook group. The ratings for item 3 ("The format of each exercise is good,") and item 4 ("The content of the exercises is good,") also show a significant difference between the two groups, in spite of the fact that both groups received the same exercises in the same format. One possible reason for these differences might be the lack of interactive features in the workbook exercises, The differences in the ratings for item 9 ("I find the exercises interes6ng,") and item 11 ("I want to practice Japanese by using this type of exercise on a regular basis,") also suggest that the students in the CALI group were more enthusiastic about the exercises.

CALI Workbook T-test

Item Mean SD Mean SD t Sig of t

1. The grammar notes are helpful. 5.0 0.0 4.5 0.7 2.82 0.01

2. The grammar notes are written clearly. 4.7 0.7 3.5 1.2 2.89 0.01

3. The format of each exercise is good. 4.9 0.3 3.8 0.8 4.53 0.001

4. The content of the exercises is good. 4.9 0.3 4.2 0.6 4.26 0.001

5. The vocabulary hints are helpful. 4.9 0.3 4.8 0.4 0.52 NS

6. I can work on the exercises at my

own pace. 4.8 0.5 4.1 1.2 1.84 NS

7. I did not get nervous while I was

working on the exercises. 4.7 0.7 4.0 1.2 1.76 NS

8. The exercises can help me practice

Japanese by myself. 4.5 0.7 3.4 1.4 2.52 0.02

9. I find the exercises interesting. 4.8 0.5 3.8 0.6 4.59 0.001

10. The exercises help me concentrate

on studying Japanese. 4.8 0.5 3.6 1.0 3.47 0.002

11. I want to practice Japanese by using

this type of exercise on a regular basis. 4.5 0.7 3.3 1.4 2.42 0.02

12. The exercises are good as

supplementary work outside of the

classroom in learning Japanese. 4.9 0.3 3.8 1.1 3.52 0.002

13. I learned a lot from the exercises. 4.9 0.3 3.8 1.0 3.87 0.001

14. The instructions in the program

are clear. 4.8 0.5 n/a n/a

15. The program is easy to use. 4.8 0.5 n/a n/a

16. I didn't have technical problems when

working on the program. 4.8 0.4 n/a n/a

17. The processing of feedback is fast

enough. 2.5 1.7 n/a n/a

18. The error messages are easy to

understand. 4.1 1.2 n/a n/a

19. The error messages are helpful in pointing

out what is wrong with my response. 4.5 0.8 n/a n/a

20. The error messages are helpful in pointing

out why my response is wrong. 4.6 0.7 n/a n/a

66

21. The error messages have the appropriate

amount of detail. 4.8 0.6 n/a n/a

22. The quality of the error messages

is good. 4.8 0.4 n/a n/a

Table 5. The questionnaire on Nihongo-CALI and the workbook instruction

In addition to the items presented in Table 5, the students were asked to write their impressions of the computer program or of the workbook they received. Most of the students in the CALI group expressed that they liked the computer program and considered it very helpful. The students in the workbook group varied in their impressions: a few students said it was helpful, good, decent, etc., but a few said they did not like it. Several students in the workbook group suggested that more explanations of the particles or more instruction from the teacher would help. Apparently, the students did not think that self-study with this type of workbook is sufficient to learn the Japanese particles and sentence production, On the other hand, the students in the CALI group did not make such comments. The only suggestion made by the CALI group was to increase the processing speed of feedback. Nihongo-CALI is programmed in both LISP and the cT language¹⁸ and the communication between the two programming languages is slow when the program runs on a personal computer such as the Mac IIci used in the study. Future research will include the improvement of the processing speed in Nihongo-CALI by reprogramming it in LISP only.

CONCLUSION

The results of the study show that given the same grammar notes and exercises, ongoing intelligent computer feedback is more effective than simple workbook answer sheets for developing learners' grammatical skill in producing Japanese particles and sentences. A significant difference between Nihongo-CALI and the workbook instruction was observed in the production tests but not in the comprehension tests. This is consistent with Flynn's hypothesis that grammatical competence is less critical in comprehension than in production. As suggested by Pederson and Dunkel, the present study also confirms that the use of a medium (i.e., computer) alone does not bring better effects; rather the quality of the messages produced by the medium affects the result. This is based on the fact that the intelligent version of Nihongo-CALI is significantly more effective than the workbook instruction but the traditional version of Nihongo-CALI does not show a significant

67

difference. The study demonstrates an effective application of natural language processing technology to second language instruction. Finally, the questionnaire results show that the students in the CALI group had significantly better attitudes towards their instruction than the students in the workbook group, even with respect to aspects of instruction that were the same for both groups.

The present study demonstrates an advantage of an intelligent CALI program over a workbook instruction. However, this does not mean that intelligent CALI is always more effective than workbook instruction. Depending on the complexity of the target structures, the kinds of tasks assigned to the students, and the learner's knowledge level, the workbook type of instruction might be as effective as intelligent CALI programs. For example, if the target structures are simple and easy to learn, if tasks involve only comprehension, or if the learners are already familiar with the structures, workbook exercises with simple answer sheets might be just as useful and might be less expensive to develop and use. Continued research concerning these variables would be required in order to determine the conditions in which the intelligent type of CALI feedback can be most effective.

NOTES

¹ The romanization used in this paper is an adaptation of the Shin-kunrei-shiki "New official system" (Jorden 1987).

² In colloquial speech, the particles, ga and o, are sometimes dropped, but since the exercises are designed to provide practice using the Japanese particles, the students are instructed not to drop any phrase particle.

³ The Nihongo-CALI system runs both on a Macintosh personal computer (with minimum five MB of RAM and five MB of disk) and on a DECworkstations.

⁴ The particles were not listed in the vocabulary hint.

⁵ The Nihongo-CALI program accepts alternative versions of the correct answer if they are appropriate, such as using a different lexical item or a different word order, inserting a sentence particle (e.g., the assertion marker, yo), and so forth. The program also accepts different romanizations such as "chi" for "ti", "shi" for "si", "sha" for "sya", "tsu" for "tu", etc.

⁶ Two students in one class obtained less than 70% on the mid-term exam and all other students obtained more than 80%. Those two students were eliminated from the experiment because they were far behind in class and their inclusion caused a significant difference between the mid-term scores of the two classes.

⁷ The following scoring system was used for the pre-test. Points were deducted according to the relative importance of errors in the given questions: 1.0 score was deducted for an incorrect or missing particle, 1.0 score for an incorrect or missing verb (although when the error concerned only the verbal conjugation such as using imperfective for perfective or negative for affirmative, only 0.5 score was deducted), and 0.5 score was deducted for an incorrect or missing word (other than incorrect/missing particles and verbs). Incorrect word order such as placing a noun phrase before a verbal predicate also resulted in a 0.5 score reduction.

68

⁸ Regarding the time spent for the computer or workbook session, the workbook group tended to finish each session earlier than the CALI group. The students in the workbook group took about 30 to 50 minutes for each workbook session, while the students in the CALI group took about 40 to 60 minutes for each computer session. This time difference might be explained by the following fact. First of all, typing answers on the computer, reading feedback messages, and turning the screen (3 to 5 pages of grammar notes and 16 to 19 pages of exercises) took more time than reading and writing in the workbook (altogether 3 to 4 pages) and correcting answers with the answer sheet. Moreover, the computer required about 10 to 35 seconds per sentence to parse the student's response and to return the appropriate feedback. Nihongo-CALI was originally developed on DEC workstations on which the average processing speed was 5 seconds for each student answer analyzed. However, the availability of DEC workstations was limited at the location of this study, so the system was implemented on Macintosh IIci's which increased the processing time. Allowing for the extra time required for computer processing, both groups spent approximately the same average amount of time working on the exercises.

⁹ The post-test followed the same scoring system used in the pre-test. On the comprehension task, 1.0 score was deducted for an incorrect translation of each sentence. When the translation is partially correct, less than 1.0 score was deducted, depending on the severity of the error.

¹⁰ The same scoring systems as used in the post-test and in the first comprehension test were applied to the retention test and the second comprehension test, respectively.

¹¹ The same scoring systems as used in the post-test were applied to the oral test.

¹² the mean score of the CALI group on the mid-term exam was 90.8 (SD=5.3) and that of the workbook group was 88.3 (SD=5.4) out of a total score of 100.

¹³ The mean score of the CALI group on the pre-test was 10.0 (SD=3.2) and that of the workbook group was 9.0 (SD=2.6) out of a total score of 22.5.

¹⁴ The present study included one more sentence-production question in the pre-test and this change made the pre-test more difficult that the pre-test in the previous study. Therefore, only scores on the fill-in-the-blank questions in the present pre-test and in the previous pre-test are compared.

¹⁵ The average post-test score of the workbook group was 44.6 (SD=9.4) and that of the previous intelligent-CALI group was 56.8 (SD=8.8) out of a total score of 71.

¹⁶ There were a few sentences in which understanding the particles was crucial for interpreting the sentence. for example, five student (2 students in the CALI group and 3 students in the workbook group) interpreted the sentence, Kono tegami wa dare ga kakimasita ka incorrectly. This sentence should be interpreted as "As for this letter, who wrote it?" but they wrote "Whom did you write this letter to?" or "Whom is this letter written to?" It appears that they only used semantic cues, kono tegami "this letter," dare "who," and kakimasita "wrote" and ignored a syntactic cue, ga as the subject marker.

¹⁷ When transcribing the students' performance on the oral production tasks, it was found that the performance of two CALI students was not recorded on the tape (probably the instructor did not notice that the tape had already ended when those students took the test). Those two students were eliminated from the analysis of the test. Also, the instructor forgot to include some of the oral tasks for one student in the workbook group, so he was dropped from the analysis of the test. Accordingly, the data from 23 students (11 in the CALI group and 12 in the workbook group) were analyzed for their oral performance. The mid-term scores (excluding those three students) indicate that there is no significant difference between the two groups in language proficiency before starting the experiment (t=0.89).

¹⁸ The cT language is a programming language and environment under development at Carnegie Mellon University (Sherwood 1988).

69

APPENDIX: THE FIRST PAGE OF THE GRAMMAR NOTES

PROVIDED BY NIHONGO-CALI

(1) Subject GA and Object O

The phrase-particles GA and O are attached to a nominal. GA and O are used with action verbs which involve human decision or will (e.g., tabemasu "eat," ikimasu "go," kaimasu "buy," tukurimasu "make," etc.) GA indicates the subject who performs the action. O indicates the object that the action operates upon. Study the following sentence.

1. Tanaka-san ga susi o tukurimasita. "Ms. Tanaka made sushi."

In the sentence (1), TANAKA-SAN is the subject who performed the action of making sushi, thus it is marked with GA. SUSI is the object that the action of making operated upon; thus it is marked with O.

The following show more examples using the subject marker GA and the object marker O.

2. Sumisu-san ga konptuutaa o tukaimasu. "Mr. Smith will use a computer."
3. Sumisu-san ga atarasii waapuro o kaimasita. "Mr. Smith bought a new word processor."
4. A: Dare ga kimasita ka. "Who came (there)."

B: Sumisu-san ga kimasita. "Miss Smith came (there)."

5. A: Nani o nomimasita ka. "What did you drink?"

B: Koohii o nomimasita. "I had coffee."

In the above sentences, the ga-marked nominals are the performers of the actions and the o-marked nominals are the objects that the actions operate upon.