专利汇可以提供3D VIDEO PROCESSING专利检索,专利查询,专利分析的服务。并且A method, an apparatus, and a non-transitory computer readable medium for performing 2D to 3D conversion are presented. A 2D input source is extracted into left and right 3D images. Motion vectors are calculated for the left and right 3D images. Frame rate conversion is performed on the left 3D image and the right 3D image, using the respective calculated motion vectors, to produce motion compensated left and right 3D images. The left and right 3D images and the motion compensated left and right 3D images are reordered for display.,下面是3D VIDEO PROCESSING专利的具体信息内容。
What is claimed is:
This application is a divisional of U.S. patent application Ser. No. 12/397,448, filed Mar. 4, 2009, the entire contents of which is hereby incorporated by reference as if fully set forth herein.
The present invention relates generally to video processing, and more particularly, to a method and apparatus for processing 3D video.
Current 3D content is based on a film source, which has a rate of 24 frame per second. Television broadcasts are at either a 50 Hz or a 60 Hz refresh rate. To convert a film source for television broadcast, there needs to be a frame rate conversion. This is typically accomplished by a frame rate conversion (FRC) process.
One way of performing FRC on a film source to a 60 Hz refresh rate is to use a 2:3 pulldown, in which a first frame is displayed twice and a second frame is displayed three times. This is repeated for each pair of frames in the film source. Because the frame rate for television is not the same as that for film, there is a motion judder problem (i.e., the viewer may notice that motion within the image is not smooth) introduced during the 2:3 pulldown.
The output of the FRC process is a new (middle) frame that is placed between successive source frames. A problem with the new frame can arise if there is missing information regarding an obscured or hidden object in the background. If there is missing information, FRC cannot accurately produce the middle frame. This is referred to as the occlusion problem. If the middle frame is produced by motion estimation (determining a motion vector) and motion compensation (applying the motion vector to reduce the effects of motion), i.e., by the FRC process, then some immaterial images may be produced. This problem exists in both 2D images and in 3D images, where the hidden material can be in the left image, the right image, or both.
An example of an occlusion region is shown in
There is therefore a need to reduce judder when converting a film source, in particular with a 3D film source. There is also a need to address the occlusion problem in 3D sources.
The judder problem can be addressed by separating a 3D image into its component left image and right image, performing FRC on the individual images, and then reordering the images for display. The separation and FRC processes can be reversed, in that FRC can be performed on the 3D image and the original and motion compensated images can be separated into their component left and right images.
One way to address the occlusion problem in 3D sources is to utilize the disparity information (the difference between the left image and the right image) to help generate the hidden information.
A more detailed understanding of the invention may be had from the following description, given by way of example, and to be understood in conjunction with the accompanying drawings, wherein:
One way to reduce the judder problem is via proper application of FRC. This can be achieved by using motion estimation and motion compensation during the FRC process. With a 3D input source, the motion estimation and motion compensation needs to be performed on both the left image and the right image of a 3D video frame. The left image and the right image need to be separated at some point during the process; the separation can occur before FRC (as shown in
It is noted that the computational complexity is approximately the same for both methods. With a 3D image, each left image and right image is half the size of a full frame. Processing (e.g., performing FRC) on a smaller size image is easier, and the method shown in
Separation, then FRC
It is further noted that only one motion vector may be calculated, for either the left image or the right image, and that motion vector can be applied to the other image. The reasoning behind this possibility is that in most cases (approximately 95% on the time), the motion vector between the left image and the right image is very similar. A slight difference between the respective motion vectors (if calculated separately) should not have an effect. If the motion vectors between the left image and the right image are substantially different, then the disparity will be different and the motion vectors should be calculated separately. By calculating only one motion vector, the hardware complexity can be reduced; the determination whether only one motion vector is calculated can be based on the hardware present and is implementation-specific.
One exceptional case to note is when an object in the image is approaching only one eye. At a distance close to the viewer, there will be a large disparity between the images, and there will also be a large difference in the motion vectors. This is because the left image and the right image have different motions associated with them, since the object is moving in a different direction and at a different distance from each eye.
FRC is then performed on the left image (step 208) and on the right image (step 210). Performing the FRC (steps 208 and 210) can also be reversed, provided that the motion vectors are determined before FRC is performed. After FRC has been performed on both the left image and the right image, the converted images are reordered for display (step 212).
FRC, Then Separation
2D to 3D extraction, then FRC
FRC is then performed on the left image (step 608) and on the right image (step 610). Performing the FRC (steps 608 and 610) can also be reversed, provided that the motion vectors are determined before FRC is performed. After FRC has been performed on both the left image and the right image, the converted images are reordered for display (step 612).
FRC, Then 2D to 3D Extraction
Apparatus Configured to Perform the Methods
The apparatus 1000 includes a 2D to 3D image extracting device 1002, a motion vector calculating device 1004, a 3D image separating device 1006, a frame rate conversion device 1008, and an image reordering device 1010.
When performing the method 200, the apparatus 1000 operates as follows. A 3D input 1020 is provided to the 3D image separating device 1006, which separates the 3D input 1020 into left and right images. Motion vectors for the left and right images are calculated by the motion vector calculating device 1004. The frame rate conversion device 1008 performs FRC on the left and right images using the calculated motion vectors to produce motion compensated images. The left and right images and the motion compensated images are passed to the image reordering device 1010, where the images are reordered for display and are output 1022.
When performing the method 400, the apparatus 1000 operates as follows. A 3D input 1030 is provided to the motion vector calculating device 1004, which calculates the motion vector for the 3D input 1030. The frame rate conversion device 1008 performs FRC on the 3D input 1030 using the calculated motion vector to produce a motion compensated 3D image. The 3D input 1030 and the motion compensated 3D image are passed to the 3D input 1030 separating device 1006, which generates left and right images from the 3D image and left and right images from the motion compensated 3D image. The original left and right images and the motion compensated left and right images are passed to the image reordering device 1010, where the images are reordered for display and are output 1022.
When performing the method 600, the apparatus 1000 operates as follows. A 2D input 1040 is provided to the 2D to 3D image extracting device 1002, which extracts the 2D input 1040 into left and right 3D images. The motion vector calculating device 1004 calculates motion vectors for the left and right images. The frame rate conversion device 1008 performs FRC on the left and right images using the calculated motion vectors to produce motion compensated left and right images. The left and right images and the motion compensated left and right images are passed to the image reordering device 1010, where the images are reordered for display and are output 1022.
When performing the method 800, the apparatus 1000 operates as follows. A 2D input 1050 is provided to the motion vector calculating device 1004, which calculates the motion vector for the 2D input 1050. The frame rate conversion device 1008 performs FRC on the 2D input 1050 using the calculated motion vector to produce a motion compensated image. The 2D input 1050 and the motion compensated image are passed to the 2D to 3D image extracting device 1002, which extracts the 2D input 1050 and the motion compensated image into left and right 3D images and produces an output 1052.
Addressing the Occlusion Problem
The occlusion problem in 3D images can be addressed by using both the left image and the right image together during FRC. The two images may contain the information missing from a 2D image due to the different information contained in the left image and the right image (also referred to as disparity information).
Generally, motion estimation in block matching can be stated as, for example, the calculation of the minimization of the mean absolute difference (MAD) between the viewer's left and right eyes (d1 and d2).
where B denotes an N1×N2 block for a set of candidate motion vectors (d1, d2) and signal s is at a pixel (n1, n2) in frame k (shown in
At time t+1, block B 1204 has moved to the right and the viewer can see more of block C 1206 with their right eye. Ideally, when the motion vector from time t to time t+1 is comparable to the disparity, the angle θ can be defined as:
where v is a motion vector, d is a disparity vector, and d* is the distance between the viewer's left eye view and the viewer's right eye view.
FRC is then performed on the 2D stream (step 1410). This step includes separately performing FRC on the left image and the right image (each a 2D stream). 2D FRC can also be used in cases where there is a 3D input, but only a 2D display, and therefore, the FRC output needs to be a 2D stream.
The present invention can be implemented in a computer program tangibly embodied in a non-transitory computer-readable storage medium containing a set of instructions for execution by a processor or a general purpose computer; and method steps can be performed by a processor executing a program of instructions by operating on input data and generating output data. Suitable processors include, by way of example, both general and special purpose processors. Typically, a processor will receive instructions and data from a read-only memory (ROM), a random access memory (RAM), and/or a storage device. Storage devices suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). In addition, while the illustrative embodiments may be implemented in computer software, the functions within the illustrative embodiments may alternatively be embodied in part or in whole using hardware components such as Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other hardware, or in some combination of hardware components and software components.
While specific embodiments of the present invention have been shown and described, many modifications and variations could be made by one skilled in the art without departing from the scope of the invention. The above description serves to illustrate and not limit the particular invention in any way.
标题 | 发布/更新时间 | 阅读量 |
---|---|---|
一种视频解码中运动补偿的方法及装置 | 2020-05-08 | 251 |
一种基于融合神经网络的视频中人体行为识别方法 | 2020-05-08 | 645 |
一种利用INSAR精确解算采煤沉陷区地表下沉量的方法 | 2020-05-08 | 940 |
一种光功率自动调节方法及其系统 | 2020-05-08 | 903 |
细化运动矢量的利用 | 2020-05-08 | 196 |
基于实时视频与规则的反馈式区域入侵检测方法与系统 | 2020-05-08 | 248 |
多频段虚拟的扩展阵列系统 | 2020-05-08 | 668 |
用于体育视频识别的快速对象跟踪框架 | 2020-05-11 | 63 |
一种基于块编码特性的3D视频错误隐藏方法 | 2020-05-08 | 451 |
适应于全向视频的运动矢量预测量的方法和装置 | 2020-05-08 | 991 |
高效检索全球专利专利汇是专利免费检索,专利查询,专利分析-国家发明专利查询检索分析平台,是提供专利分析,专利查询,专利检索等数据服务功能的知识产权数据服务商。
我们的产品包含105个国家的1.26亿组数据,免费查、免费专利分析。
专利汇分析报告产品可以对行业情报数据进行梳理分析,涉及维度包括行业专利基本状况分析、地域分析、技术分析、发明人分析、申请人分析、专利权人分析、失效分析、核心专利分析、法律分析、研发重点分析、企业专利处境分析、技术处境分析、专利寿命分析、企业定位分析、引证分析等超过60个分析角度,系统通过AI智能系统对图表进行解读,只需1分钟,一键生成行业专利分析报告。