EYEGLASS INTERFACE SYSTEM |
|||||||
申请号 | EP98955183.3 | 申请日 | 1998-10-29 | 公开(公告)号 | EP1027627B1 | 公开(公告)日 | 2009-02-11 |
申请人 | MYVU Corporation; | 发明人 | SPITZER, Mark, B.; | ||||
摘要 | An eyeglass interface system is provided which integrates interface systems within eyewear. The system includes a display assembly and one or more audio and/or video assemblies mounted to an eyeglass frame. The display assembly is mounted to one temple and provides an image which can be viewed by the user. The audio or video assembly is mounted to the other temple and is in communication with the display assembly. The audio or video assembly may comprise a camera assembly and/or an audio input or output assembly, such as a microphone and/or speakers. The camera assembly is placed on the temple to record the visual field observed by the user. A head-tracking assembly may be provided to track the position of the user's head. A number of applications can be provided with the present system, such as a telephone system, pager system, or surveillance system. The present eyeglass interface system is compact, offers the user hands-free operation, and provides an attractive appearance due to concealment of the assemblies within the eyeglass frame and lenses. | ||||||
权利要求 | |||||||
说明书全文 | Information storage and processing, electronic recording of sound and images, electronic communications, and electronic entertainment systems have become widespread, and portable applications of these technologies are growing rapidly. The integration of a camera with a head-mounted display was disclosed by Hawthorne in In In a publication entitled " Document Full portability for many of these interface systems requires reduction to the portable scale of the user interface. For most applications, it is desirable to have an audio/video interface which is compact and allows hands-free or nearly hands-free operation. In addition, it is desirable that such an interface be unobtrusive and not detract from the appearance of the user. The eyeglass interface system of present invention addresses the need for such an interface, by integrating the complete audio/video interface within eyewear. The eyeglass interface system uses an eyeglass optical system or display assembly such as that in accordance with In a typical configuration according to the present invention, the display assembly is mounted within one half of the eyeglasses. The audio and/or video assembly comprises a camera assembly mounted within the other half. The system may also include audio transducers (microphone and/or speakers) which provide audio data and/or which collect sound for recording, digital signal processing and analysis. In this way, the microphone may also provide voice activated commands to the system. Additionally, the eyeglass system may include batteries and an RF or infrared communication system to eliminate the need for a cable to the glasses. Various types of systems may be formed using one or more of the subsystems. In its most complete form, the present eyeglass interface system includes the following subsystems or assemblies: an audio input assembly for collecting speech input from the user, a display assembly, an audio output assembly, a camera assembly, and a head-tracking assembly to provide a computer with the position of the user's head. Thus, the eyeglass interface system includes audio, video, tracking and power modules. The eyeglass interface system of the present invention provides computer input and output in highly compact forms, preferably entirely within eyeglasses, which look to the casual observer like ordinary eyewear. In some embodiments of the invention, the subsystems or assemblies are entirely concealed within the eyeglass frame and lenses. The invention comprises not only the integration of the hardware subsystems within an eyeglass frame, but also the integration of the subsystems with hardware and software intended for specific functions. The present invention is an improvement over devices in the prior art owing to the complete integration of the camera assembly within the eyeglass frame and lens system of the eyeglasses. The mechanical aspects of the camera, including focusing mechanisms, are integrated within the eyeglass temple, as are the electronics in the form of a multi-chip module integrated in the temple. Additionally, lenses and optical surfaces of the camera assembly are integrated in the temple and even in the eyeglass lens itself. In this way, the camera assembly may attain its most compact form, may be made light in weight, and may attain a greater degree of concealment and functionality than prior art devices. In addition, the present invention places the field of view of the camera in proximity to or convergent with the operator's field of view, so that the camera records the visual field observed by the user. The present invention allows the camera to have the same line of sight as the user's eye, without parallax error. The present invention also includes the use of the camera assembly to control a computer. While in prior art camera systems, the camera merely collects data for the computer, in the present invention, the camera assembly also collects input commands. These commands may be in the form of written symbols; hand signals, bar codes, or other forms that cause the computer to execute instructions. The integration of the camera and display within eyewear according to the present invention provides eyewear that is more compact and therefore easier to wear than prior art devices, and is more aesthetically appealing, because it looks like eyeglasses. The present invention uses optics embedded in the eyeglass lens rather than suspending them in a mount, making the system more durable. The present invention incorporates audio and video inputs as well as outputs. The audio assembly may be integrated into the eyeglass frame. Details of the visual display are not used in conjunction with normal hearing, and therefore one embodiment of the present system integrating an audio assembly seeks to avoid attenuating external sounds. The current system allows the user to interact with computer generated or other electronically relayed information without requiring the occlusion of the actual surroundings. The invention will be more fully understood from the following detailed description taken in conjunction with the accompanying drawings in which:
The integration of a camera assembly within eyewear is important for a number of applications, including image acquisition, data input by character recognition, bar code scanning, face recognition, and the like. The key factors which must be addressed in order to integrate a camera within eyewear are (i) portability, and (ii) making the camera adjustable so that it is aimed precisely where the user is looking with his or her eyes. This second factor is key for data input by character recognition. For example, if one is looking at a notepad and writing characters, the camera can be positioned within the eyewear so that the notepad is within the field of view of the camera. The characters written by the user can then be photographed by the camera and fed to a character recognition processor. Such a system can replace the keyboard as the data input device for portable applications. Currently, such functions are provided by touch sensitive screens, such as on the Palm Pilot and other pen-based data entry devices. A camera has the distinct advantage that it can work with ordinary pencil and paper. A scanner may also be built from a camera mounted within eyewear. Text that the user reads can be automatically fed to the computer, recognized, and stored. The camera's field of view is obtained through the lens 280 held within frame front 200. This facilitates concealment of the camera. Further concealment may be obtained by tinting the lens, or by using polarization coatings, or by other means known in the art. If concealment is not desired, the camera may be mounted on the outside of the temple. It should also be recognized that the camera may be mounted on either the left side (as shown in An eyeglass system in which the camera is more nearly aligned to the user's direction of gaze is shown in A camera system may be integrated within the eyeglass temple in a further manner, as shown in Another embodiment which includes an auto focus mechanism is shown in A bar code reading system may be built using the integrated system. It comprises eyeglass optics, display and camera as shown in Another application of a similar nature involves face recognition, wherein the bar code shown in In all of the above applications, it is necessary for the camera to record the view that the user has. The advantage of a camera mounted in eyewear is obtained, because the camera is aimed at the user's subject by virtue of its alignment within the eyeglasses. As noted above, the subsystems may be integrated with displays. Many important devices can be constructed with this simple arrangement. If the camera sensor comprises a long wavelength infrared focal plane array or other sensor, then the device allows the user to have vision in the long wavelength range which is well suited for vision through smoke. Firefighters use long wavelength sensors to find people in smoke-filled buildings in which the performance of ordinary vision is poor. Additionally, systems of this type can be configured for enhanced night vision by using infrared image sensors, or by using highly sensitive low-light image sensors. The system can also serve in surveillance applications in which imagery from one set of eyeglasses is transmitted to a second set. This allows a surveillance team to share images. The eyeglass embodiment of the various camera systems comprises an improvement over the prior art alternatives owing to its light weight and compact nature, as well as owing to the automatic aiming of the camera by virtue of its mounting within eyewear, and its use of the eyeglass lens to align the camera to the user's pupil. Additionally, such systems can be used without employing the hands, making it possible to conduct other tasks while using the device. The present invention also utilizes audio transducers integrated within eyeglasses as communication devices and as input/output devices for computers. In addition to the integration of audio with an eyeglass display and/or camera system, another embodiment contemplates the use of bone conductivity transducers to allow the transducers to be concealed within the eyeglass frames. Bone conductivity transducers are designed to transmit sound vibrations directly to solid media (bone) and not to develop a pressure wave in air. Accordingly, bone conductivity transducers are not generally audible unless in contact with the head. Additionally, bone conductivity transducers are not worn in the ear canal and accordingly do not block normal hearing and further do not require an audio microphone near the mouth. For this reason, bone conductivity transducers permit the concealment of the transducers within eyewear. As shown in One or more bone conduction microphones may be mounted within the eyeglass frame, including within the nose piece, to sense the vibration produced by the user's speech. Alternative positions for bone conduction microphones are along the temple or at the back of a temple behind the ear. In some cases the microphone may be mounted in an attached earpiece which may be inserted into the ear. Alternatively a directional air conduction audio microphone may be used by placing it at the bottom of the eyeglass frame oriented so as to preferentially detect the sound of the user's speech. A plurality of microphones may be used in order to make possible the implementation of noise reduction techniques using analog or digital signal processing. For certain applications, it is desirable to sense ambient sound instead of or in addition to the user's speech. In that case, an air conduction microphone similar to ones used in hearing aids may be used in any convenient place along the eyeglass frame. For the particular application of assisting the hearing impaired, the microphone would be so oriented so that sound coming from the area in front of the user is preferentially detected, in alignment with the camera direction if one is used. Alternatively, an external microphone may be used for this application. In order to accommodate the integrated circuits and flexible interconnect circuitry within the eyeglass temple, the eyeglass temple may be made hollow so as to house the circuits. Additionally, if the temple is made from metal or is metal coated, the circuits can be shielded so that the radio frequency interference from the internal circuits is minimized. Although flexible Kapton circuitry may be used as a substrate and interconnect for the internal electronics, alternatively it is possible to design a multichip module (MCM) and package using the eyeglass temple as a housing, as illustrated in The interconnection of the subsystems to external electronics may be made by individual wires; however, an improved method comprises multiplexing and demultiplexing of the signals in digital or mixed signal form. The subsystems described above are connected to remote electronics either by wire as noted above, or by infrared link, radio frequency methods, or by fiber optical cable. In each of the above embodiments, a battery 925 may also be installed in the eyeglasses which powers the interface circuits and all other devices in the eyewear. The battery may be located within the temple behind the ear or remote from the eyeglass frame and in communication with the eyeglasses by wire. The use of an IR system is illustrated in The integration of microphones, speakers, camera and display within or upon the eyeglass frames makes possible a wide range of communication and recording devices, in addition to computer input and output devices. A multi-media interface module, comprising one, several, or all of the audio and video subsystems, has numerous applications when combined with the appropriate circuitry. An application of a display system using audio within the glasses is in a hands-free telephone interface. In such an application, the eyeglass system is connected to telephone circuitry, such as a cellular or cordless telephone, provided in the module 1001. A keypad is unnecessary if voice recognition is employed. In such a case, the user speaks the command normally input by keypad, such as the number to dial, the send command, and so forth. The display within the interface module shows the number to be called for verification, just as the display on a cell phone would show the number, and after the appropriate send command is spoken, the call is placed. The user speaks and listens through the audio capability of the eyeglass system. This is an improvement over the standard cell phone interface because the user's hands are free, and the user does not have to look away from his current field of view in order to verify the number to be called. An electronic hands-free pager may be incorporated within the eyewear interface system by interconnecting the system to a pager receiver by wire within strap 1002 to module 1001. Alternatively, the pager receiver or transceiver circuit may be incorporated within the eyeglass frame itself. The interface system may be provided with a timepiece module within the eyeglass frame so that the time is displayed in the user's field of view by the display. In addition to providing the time in a hands-free manner, a further improvement over wristwatches and wall clocks is that the user may check the time in a non-obvious way thereby avoiding giving offense to others with which the user is conversing. The interface system may be connected to an information sending unit or may be fitted with an RF receiver for reception of critical information, located within module 1001. This information system may be, but is not limited to, a general function computer such as a notebook computer. For example, a receiver may be mounted in the eyeglass frame to display simple alpha-numeric characters. A transmitter in communication with this receiver may be provided with a blood pressure or other vital sign monitoring device, so that a surgeon may be provided with vital sign data for a patient. In this way the data is available to the surgeon in a hands-free manner and without the need for the surgeon to look away from the incision. Many other similar information systems are possible. For example, an electronics engineer may be provided with a display of the output of an oscilloscope or network analyzer. In this way, the engineer can see the output of the measurement device without the need to look away from the device under test. The camera and microphone within the glasses allow the system to perform many of the functions of a camcorder or still camera. Audio and video may be recorded in electronic memory and then may be transmitted to tape or disk storage media. The advantages over current camcorders are that the interface system, including the camera, is largely hands-free. The only requirement is that the user activate the recorder either by spoken command or by the use of buttons on the recording device. Ahead tracking position sensor can be used to stabilize the image. In such a device, the recording device (for example, tape recorder, recordable digital video disk system, recordable compact disk system) are located in module 1001. The camera system comprises standard CCD or CMOS image sensors that are commercially available from Texas Instruments, Kodak, and other sources. These sensors are used in camcorders and electronic cameras. The image sensors are combined with the eyeglass optics so that the camera has a view of the world as seen by the user's eye. The camera may be mounted in the same optical system as the display, or as shown in The head tracking module is useful for augmented reality applications in which information displayed by the interface system is superimposed on the user's field of view by the optical system. For example, computers are used to calculate position from global positioning satellites to provide navigation information to a user. If the system also has information on the direction of gaze of the user, by virtue of a head-tracker sensor, symbols may be provided that are registered to the field of view, such as street names. When the user looks toward a side street, the computer may calculate the position of the intersection of the streets and present the name of the crossing street to the user. Many other applications of augmented reality are possible. The tracking sensors may comprise either magnetic and gravity sensors that determine position from the earth's magnetic and gravitational fields, or from other artificial magnetic fields. Sensors may be mounted in module 1001 if it is held against the head or alternatively within the eyeglass frame. The magnetic sensor may be based on the Hall effect or may use flux gate magnetometers. Any sensor that detects magnetic field strength and direction may be used in this application, provided the sensor is small enough to fit within an eyeglass frame or module. An alternative is to position magnetic sensors within module 1001 which is worn upon the shoulders of the user, and to place small magnets within the glasses frame. In this way the magnetic sensors can determine the user's head position. Many other types of sensors may be used for the head tracking system, including miniature gyroscopic or other inertial sensors, ultrasonic sensors, and even systems that use images from the camera system to detect head movement. The eyeglass module can be used for video conferencing. In this application, the interface system comprises a display, camera, associated optics, and audio system in communication, either by wire or local RF, with a transceiver which is itself in communication with the video conferencing partner. Video conferencing systems with standard CRT video and microphone/speaker audio are known in the art. In the present invention, the user receives information in audio and video form through the system interface (display and audio subsystems) and transmits information from the system interface, this information being collected from the microphone and camera. In a video conference, the user transmits to the conference images he sees. Alternatively, the user may face a mirror to transmit his own image. The improvement over the prior art is the high degree of portability offered by this invention. Many surveillance systems are possible using the interface system, including systems in which the surveillance is concealed by the eyeglass frame. The system may be used by law enforcement personnel to transmit audio and video between agents, in a manner similar to the video conferencing application. For example, an agent on the street may see an image collected by an agent on a roof top. Another application involves patient monitoring in hospital intensive care units. A third surveillance application comprises a baby monitor. The glasses may receive images and audio from a camera located in a nursery so that the parent may see and hear the baby from anywhere in the house. The interface system may be applied in telepresence systems, in which the user receives visual and audio information from a remote source, as if he were present at the remote source. A transmitter at the remote source collects information from the environment and sends it to the receiver. As shown in The invention is not to be limited by what has been particularly shown and described, except as indicated by the appended claims. |