BACKGROUND OF THE INVENTION

1. Field of Invention

This invention relates to an apparatus and method for automatically verifying communication software, and more particularly, to an automatic verification apparatus and method which can automatically execute a regression test for communication software.

2. Description of the Related Art

FIG. 26

schematically illustrates a conventional communication system

200

. The communication system

200

has a CPU

210

, a plurality of terminal connectors

220

, and a communication software

230

, as major elements. Each of the terminal connectors

220

converts electric signals transmitted to and from the associated terminal

240

into digital signals, and supplies the digital signals to the CPU

210

. The CPU controls each terminal

240

according to the communication software

230

. Two or more terminal connectors

220

are connected to a common CPU

210

. The number of the terminal connectors

220

is determined depending on the type and the number of the terminals

240

connected to the terminal connectors

220

.

In developing a communication system, it is common to modify the existing system by, for example, adding a new function, rather than to develop a totally new system. Whenever the existing functions are modified, or new functions are added to the existing functions are modified, or new functions are added to the existing communication software, it is essential to verify or recheck the items or elements which were already tested in the original software. For example, the communication system

200

illustrated in

FIG. 26

was already subjected to a test A for operating each terminal

240

with a procedure A. If the communication software

230

is modified, it must be verified if the modified communication system

200

correctly operates with the procedure A, with which the pre-modified system

200

operates. Therefore, each terminal

240

must be test-operated with the same procedure A in order to reconfirm that the improved communication system

200

operates correctly. The process for verifying the operation result is called a “regression test”. The efficiency of the development of a communication system greatly depends on how efficiently the regression test is performed.

There are some problems in the conventional process for developing a communication system (including hardware). These problems may adversely affect the efficiency of the regression test. One major problem is that the modified communication software can not be tested unless there is a communication hardware in the test environment. Another problem is that the test is performed only in the vicinity of the hardware. Therefore, there is a strong demand for a technique which allows the communication software to be tested without hardware, or to be tested from a remote place via networks.

In addition, since it is inefficient to manually repeat the same test operation for each terminal every time a modification is made to the existing communication system, a demand for a technique for automatically repeating the test operation for each terminal has also arisen.

SUMMARY OF THE INVENTION

Therefore, it is an object of the invention to provide an automatic verification apparatus which can automatically execute a regression test for communication software. The automatic verification apparatus comprises a virtual terminal manager for performing a pseudo-operation of a terminal connector, a virtual terminal performing a pseudo-operation of a terminal, a scenario recording unit for recording the pseudo-operation performed by the virtual terminal as a scenario, a scenario executing unit for automatically executing the recorded pseudo-operation of the terminal, and a scenario verification unit for automatically verifying if the result of the automatic execution is correct whenever the recorded pseudo-operation of the terminal is automatically executed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1

is a block diagram for an automatic verification apparatus for communication software.

FIG. 2

is a block diagram of the test execution unit shown in FIG.

1

.

FIG. 3

is a block diagram of the memory shown in FIG.

2

.

FIG. 4

illustrates the structure of the software of the test execution unit.

FIG. 5

is a block diagram of the hard drive.

FIG.

6

(

a

) illustrates a typical message structure, and

FIG.

6

(

b

) illustrates a verification message.

FIG. 7

illustrates the structure of a scenario recording unit.

FIG. 8

illustrates the structure of a scenario verification unit.

FIG. 9

the structure of a scenario executing unit.

FIG.

10

(

a

) illustrates a message structure, and

FIG.

10

(

b

) is a sequence diagram showing the sequence of the message shown in FIG.

10

(

a

) in time series.

FIG.

11

(

a

) illustrates a message sequence recorded as a correct message order for executing a test, and

FIG.

11

(

b

) illustrates a message sequence of the execution result of an automatic verification.

FIG.

12

(

a

) illustrates a message group file, and

FIG.

12

(

b

) illustrates how the message group shown in FIG.

12

(

a

) are recorded in the scenario recording unit.

FIG. 13

is a flowchart showing the sequence of the group data preparation procedure.

FIG. 14

shows the recording sequence of the scenario recording unit.

FIG. 15

is a flowchart showing the sequence of the scenario recording procedure.

FIG. 16

illustrates the scenario verification unit.

FIG. 17

is a flowchart showing the sequence of the verification preparation procedure.

FIG. 18

illustrates the state of the scenario verification unit after message E

1

has been transmitted.

FIG. 19

illustrates the state of the scenario verification unit after message O

3

has been received.

FIG. 20

illustrates the state of the scenario verification unit after message O

1

has been received.

FIG. 21

illustrates the state of the scenario verification unit after message E

2

has been received.

FIG. 22

illustrates the state of the scenario verification unit after message O

4

has been received.

FIG. 23

illustrates the state of the scenario verification unit after message O

2

has been received.

FIGS. 24 and 25

are flowcharts showing the sequence of the message verification procedure.

FIG. 26

illustrates a conventional communication apparatus.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The preferred embodiments of an automatic verification apparatus and method for a communication software according to the invention will now be described with reference to the attached drawings.

First, the functional structure of the automatic verification apparatus for a communication software according to an embodiment will be explained. The automatic verification apparatus comprises (1) a functional element for performing a pseudo-operation of a terminal connector, (2) a functional element for performing a pseudo-operation of a terminal, (3) a functional element for recording the pseudo-operation performed by the virtual terminal as a scenario, (4) a functional element for automatically executing the recorded pseudo-operation of the terminal, and (5) a functional element for automatically verifying if the automatic execution result is correct whenever the recorded pseudo-operation of the terminal is automatically executed. In this specification, a pseudo terminal connector is referred to as a “virtual terminal manager”, a pseudo terminal is referred to as a “virtual terminal”, and a file in which the operation of the virtual terminal is recorded is referred to as a “scenario”.

Next, the actual hardware and software structures representing the above-described functional elements of the automatic verification apparatus

100

for a communication software will be explained with reference to

FIGS. 1 through 5

.

FIG. 1

illustrates the automatic verification apparatus

100

according to an embodiment of the invention.

This automatic verification apparatus

100

has two major parts, a test object communication apparatus

1

and a test execution apparatus

2

. The test object communication apparatus

1

and the test execution apparatus

2

are connected to each other via the network interface

5

provided to the apparatus

1

and the network interface provided to the test execution unit

6

in the apparatus

2

.

The test object communication apparatus

1

comprises communication software

3

, a CPU

4

for controlling the communication software

3

, and a network interface

5

which functions as an input/output unit to and from the test execution apparatus

2

. This test object communication apparatus

1

is similar to the conventional communication apparatus

200

illustrated in

FIG. 26

, but the terminal connector

220

of the conventional communication apparatus

200

is replaced by the network interface

5

. The object of the test communication apparatus

1

is the communication software

3

. The terminal connector

220

shown in

FIG. 26

is not the test object in this invention, the communication software

3

can be correctly tested even if the terminal connector, which is the actual hardware, is not connected to the test object communication apparatus

1

.

The test execution apparatus

2

comprises a test execution unit

6

, an input unit consisting of a keyboard

7

and a mouse

8

, through which an operator can externally operate the apparatus

2

, and display

9

for displaying the contents of the operation input through the keyboard

7

and the mouse

8

. The keyboard

7

, the mouse

8

, and the display

9

are commercially available, and the structures of these elements are known. Therefore, explanation for these elements is omitted.

The test execution unit

6

is essential for the automatic verification apparatus

100

.

FIG. 2

illustrates the structure of the test execution unit

6

. In the test execution unit

6

, a CPU

11

for processing data, a memory

12

for storing data, a timer

13

for counting processing time, a time-out clock

14

, a hard drive

15

for storing a large volume of data, a keyboard controller

16

for controlling the keyboard

7

, a mouse controller

17

for controlling the mouse

8

, a video controller

18

for controlling the display

9

, and a network interface

19

, which functions as an input/output unit to and from the test object communication apparatus

1

, are mutually connected via a data-transfer bus

10

.

The elements other than the memory

12

and the hard drive

15

are commercially available, and known to those skilled in the art. The software structure stored in the memory

12

, and the file structure stored in the hard drive

15

are essential for the automatic verification apparatus

100

, the details of which will be described below.

FIG. 3

illustrates the structure of the memory

12

. The memory

12

comprises a memory device

27

which stores test execution software

28

, and a commercially available memory controller

26

for controlling the memory device

27

. The memory device

27

and the test execution software

28

stored in the memory device

27

are essential for the automatic verification apparatus

100

for communication software.

FIG. 4

shows the structure of the test execution software

28

, which has a virtual terminal manager

30

, which is a pseudo terminal connector formed for a test, and a scenario manager

31

for managing operation scenarios of the virtual terminal. The scenario manager

31

also manages a scenario recording unit

32

for recording scenarios, scenario executing unit

33

for executing the scenario, and a scenario verification unit

34

for verifying the scenario. The scenario manager

31

can access the hard drive

15

if it is necessary.

FIG. 5

shows the structure of the hard drive

15

. The hard drive

15

comprises a hard drive controller

20

and a hard drive medium

21

. The hard drive medium

21

stores a directory

22

, and a plurality of files

23

. These files

23

stored in the hard drive medium

21

can be identified by their unique names. The list of the file names is registered in the directory

22

. The CPU

11

can access each of the files

23

, via the bus

10

and the hard drive controller

20

, by identifying the file name. It is also possible to read out and write the data from and into each file by byte, as well as to execute ordinary file manipulation, such as file creation or deletion. Each file

23

consists of a message group file

58

for storing data as to the message groups, the details of which will be described below, and a scenario file

59

for storing data as to the scenarios.

In

FIG. 4

, the solid lines and the broken lines connected among the virtual terminal manager

30

, the scenario manager

31

, the scenario recording unit

32

, and the scenario verification unit

34

indicate the data flow. The data transferred along the solid lines are “messages”, which are communicated from the test object communication unit

1

via the network interface

19

. The data transferred along the broken lines are “control data”, which are communicated among the display

9

, the keyboard

7

, and the mouse

8

in order to control the virtual terminal manager

30

and the scenario manager

31

. The display

9

displays an operation panel (for example, a virtual terminal

29

) for operating the virtual terminal manager

30

and the scenario manager

31

. The operator can transmit control data for controlling the virtual terminal manager

30

and the scenario manager

31

to the bus

10

using the keyboard

7

and the mouse

8

, while referring to the operation panel displayed on the display

9

.

FIG. 6

shows the structure of the “message” communicated between the communication software

3

and the virtual terminal manager

30

. There are two major types of messages, an ordinary message and a verification message. FIG.

6

(

a

) shows the structure of the ordinary message

35

, and FIG.

6

(

b

) shows the structure of the verification message

39

.

The ordinary message

35

consists of a transmitter terminal ID

36

, a receiver terminal ID

37

, and a message ID

38

, as shown in FIG.

6

(

a

). If a scenario is recorded by the scenario recording unit

32

, the message

35

is recorded in the format of the verification message

39

. This will be explained below. A verification message

39

includes a transmitter terminal ID

40

, a receiver terminal ID

41

, a message ID

42

, a group ID

43

, and time

44

, as shown in FIG.

6

(

b

). Thus, the verification message

39

has a group ID

43

and time

44

, in addition to the contents of the ordinary data

35

.

FIGS. 7 through 9

show the structures of the scenario recording unit

32

, the scenario verification unit

34

, and the scenario executing unit

33

, respectively, which are all administrated by scenario manager

31

of the test execution software

28

.

As shown in

FIG. 7

, the scenario recording unit

32

has a message list

45

and a group list

46

. The message list

45

includes a plurality of verification messages

39

, which are arranged in chronological order. Therefore, the newest message

35

which was communicated most recently, is recorded in the form of a verification message

39

placed last in the list.

The group list

46

includes a plurality of group-list elements

47

, each group-list element consisting of a plurality of attached message elements

50

which are recorded in the corresponding attached message list

49

. Each group-list element

47

can store a group name or group ID. Similarly, each attached message element

50

can have a message name or message ID. These group-list elements

47

and the attached message elements

50

are arranged in chronological order. That is, the newest message

35

, which was communicated most recently, is added to the last of the group-list elements

47

and the attached message elements

50

.

FIG. 8

shows the structure of the scenario verification unit

34

. The scenario verification unit

34

has a terminal list

52

. The terminal list

52

includes a plurality of terminal-list elements

53

, each corresponding to one of the terminals. Each terminal-list element

53

includes a plurality of verification messages

39

which are managed by the message list

55

provided to each terminal-list element

53

. Each terminal-list element

53

can store the terminal ID. Both the terminal-list elements

53

and the verification messages

39

are arranged in chronological order, and the newest message

35

is added to the end of the message list, as a verification message

39

, for each terminal-list element

53

.

FIG. 9

shows the structure of the scenario executing unit

33

. The scenario executing unit

33

has a line task buffer

56

and a character string task buffer

57

. The line task buffer

56

can store a line of a message group of the message group file

58

, and the character string task buffer

57

can store a character string of the message group.

The structure of the test execution software

28

applicable to the automatic verification apparatus

100

for communication software according to the invention has been described above. The operations of the test execution software

28

will now be explained.

First, the message sequence is explained with reference to FIG.

10

. FIG.

10

(

a

) illustrates the detailed structure of the message

35

shown in FIG.

6

(

a

). The messages

35

E

1

,

35

O

1

,

35

E

1

, and

35

O

2

are transferred between the virtual terminal manager

30

and the communication software

3

via the network interface

5

of the test object communication unit

1

and the network interface

19

of the test execution unit

6

.

FIG. 10

(

b

) shows the message flow of these messages

35

E

1

,

35

O

1

,

35

E

1

, and

35

O

2

in time series. In this specification, the message flow depicted in time series, as in FIG.

10

(

b

), is called a “message sequence”.

FIG.

10

(

a

) shows four messages

35

E

1

,

35

O

1

,

35

E

2

, and

35

O

2

. The message flows of these four messages are depicted in chronological order, and the latest message flow between the communication software

3

and the virtual terminal manager

30

is positioned at the bottom of the sequence.

The first message

35

E

1

is transmitted from “Term

1

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, to the communication software

3

, which is the receiving terminal indicated as “SYS” in FIG.

10

(

b

), via the network interfaces

19

and

5

. In this specification, the message transmitted from the virtual terminal manager

30

to the communication software

3

is called an “event”, and the notation “E

1

” denotes the first event.

The second message

35

O

1

is transmitted from the communication software

3

, which is the receiving terminal indicated as “SYS”, to “Term

1

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, via the network interfaces

5

and

19

. In this specification, the message transmitted from the communication software

3

to the virtual terminal manager

30

is called an “order”, and the notation “O

1

” denotes the first order.

The third message

35

E

2

is a message for transmitting an event E

2

from “Term

2

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, to the communication software

3

, which is the receiving terminal indicated as “SYS”, via the network interfaces

19

and

5

.

The fourth message

35

O

2

is a message for transmitting an order O

2

from the communication software

3

, which is the receiving terminal indicated as “SYS”, to “Term

2

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, via the network interfaces

5

and

19

.

In FIG.

10

(

b

), the vertical lines are time axes, and the horizontal message flows are indicated as arrows. As is clear from the message sequence of FIG.

10

(

b

), event E

1

is transmitted as the message

35

E

1

from the virtual terminal Term

1

to the communication software

3

(“SYS”), then order O

1

is transmitted as the message

35

O

1

from the communication software

3

(“SYS”) to the virtual terminal Term

1

. Then, event E

2

is transmitted as the message

35

E

2

from the virtual terminal Term

2

to the communication software

3

(“SYS”), and finally, order O

2

is transmitted as the message

35

O

2

from the communication software

3

(“SYS”) to the virtual terminal Term

2

.

Next, each of the elements of the automatic verification unit for communication software according to an embodiment of the invention will be described in detail. The elements are essentially (1) a functional element for performing a pseudo-operation of a terminal connector, (2) a functional element for performing a pseudo-operation of a terminal, (3) a functional element for recording the pseudo-operation performed by the virtual terminal as a scenario, (4) a functional element for automatically executing the recorded pseudo-operation of the terminal, and (5) a functional element for automatically verifying if the automatic execution result is correct whenever the recorded pseudo-operation of the terminal is automatically executed.

The functional element for performing a pseudo-operation of a terminal connector is embodied as a virtual terminal manager

30

of the test execution software

28

. The functional element for performing a pseudo-operation of a terminal is embodied as a virtual terminal

29

. The functional element for recording the pseudo-operation of the virtual terminal is embodied as group data preparation procedure and scenario recording procedure executed by the scenario manager

31

and the scenario recording unit

32

. The functional elements (4) and (5) are simultaneously embodied as a verification preparation procedure and a message verification procedure executed by the scenario manager

31

, the scenario executing unit

33

, and the scenario verification unit

34

.

FIG.

11

(

a

) shows a message sequence recorded as a correct message sequence for performing a regression test, and FIG.

11

(

b

) shows the actual test result.

In FIG.

11

(

a

), the first message

35

E

1

is an event E

1

transmitted from “Term

1

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, to the communication software

3

(“SYS,”) via the network interfaces

19

and

5

.

The second message

35

O

1

is an order O

1

transmitted from the communication software

3

(“SYS”) to “Term

1

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, via the network interfaces

5

and

19

. The third message

35

O

2

is an order O

2

transmitted from the communication software

3

(“SYS”) to “Term

1

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, via the network interfaces

5

and

19

.

The fourth message

35

O

3

is an order O

3

transmitted from the communication software

3

(“SYS”) to “Term

1

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, via the network interfaces

5

and

19

.

The fifth message

35

E

2

is an event E

1

transmitted from “Term

2

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, to the communication software

3

(“SYS”), via the network interfaces

19

and

5

.

The sixth message

35

O

4

is an order O

4

transmitted from the communication software

3

(“SYS”) to “Term

2

”, which is one of the virtual terminals

29

belonging to the virtual terminal manager

30

, via the network interfaces

5

and

19

.

FIG.

11

(

b

) shows the message sequence of the test result of the automatic verification. If the messages

35

E

1

and

35

E

2

are transmitted to the communication software

3

at the same timings as those of the scenarios recorded in the scenario file

59

, the order of the messages becomes the message sequence shown in FIG.

11

(

b

), and this result is determined to be a correct verification result. The reason for it will be explained below.

With the present invention, in order to obtain the determination that “the verification result is correct” from the message sequence shown in FIG.

11

(

b

), a certain scenario verification procedure must be followed. The prerequisites of the scenario verification procedure are as follows: (

a

) The transmission terminal ID and the receiving terminal ID are specified; (

b

) The message ID is specified; and (c) The group ID is specified. As has been described, “SYS” denotes the communication software

3

, and if a message is transmitted from a virtual terminal to the “SYS”, one or more messages are transmitted to the terminal from the “SYS”.

The scenario verification procedure comprises the following four procedures.

(1) Group data preparation procedure;

(2) Scenario recording procedure;

(3) Verification preparation procedure; and

(4) Message verification procedure.

It must be noted that when automatically verifying the execution result of the scenario, it cannot be verified only based on if the message sequence recorded in the scenario is the same as the message sequence resulting from the execution of the scenario. For example, if a plurality of terminals are involved in the automatic verification, the message sequence in the entire network may differ from the scenario even though the message sequence of the test result coincides with the scenario in each terminal. Furthermore, even with a single terminal, the resultant message sequence may differ from the scenario depending on the specification of the hardware and the operating system of the communication apparatus. Therefore, in this invention, the message group file shown in FIG.

12

(

a

) is used as the standard system specification.

As shown in FIG.

12

(

a

), in the message group file

58

, messages O

1

and O

2

belong to group GRP

1

, and message O

3

belongs to group GRP

2

. Since messages O

1

and O

2

belong to the same message group GRP

1

, the time order of these two messages cannot be inverted. However, the time order of one of these messages in GRP

1

and the message

35

O

3

belonging the different group GRP

2

may be inverted.

Next, the scenario verification procedure will be explained in more detail.

(1) Group Data Preparation Procedure

This procedure will be explained with reference to

FIGS. 12 and 13

. The group data preparation procedure is the process for recording the message group having the system specification shown in FIG.

12

(

a

) in the scenario recording unit

32

in a form of the group list

46

shown in FIG.

12

(

b

).

FIG. 13

is a flowchart of the group data preparation procedure. First, it is determined if the message group can be read out from the message group file

58

(S

101

). If the message group file

58

is readable, a line of message group is read out, and stored in the line task buffer

56

(S

102

). If the message group file

58

is not readable, the process terminates.

For example, if the message group

58

a

shown in FIG.

12

(

a

) is read out from the message group file

58

, the message group

58

a

has a group ID of GRP

1

, and two attached messages O

1

and O

2

. This message group

58

a

read out from the file

58

is stored in the line task buffer

56

(S