Research Article: Degraded image enhancement by image dehazing and Directional Filter Banks using Depth Image based Rendering for future free-view 3D-TV

Date Published: May 23, 2019

Publisher: Public Library of Science

Author(s): Imran Uddin Afridi, Tariq Bashir, Hasan Ali Khattak, Tariq Mahmood Khan, Muhammad Imran, You Yang.


DIBR-3D technology has evolved over the past few years with the demands of consumers increasing in recent times for future free-view 3D videos on their home televisions. The main issue in 3D technology is the lack of 3D content available to watch using the traditional TV systems. Although, some sophisticated devices like stereoscopic cameras have been used to fill the gap between the 3D content demand and 3D content supply. But the content generated through these sophisticated devices can not be displayed on the traditional TV systems, so there needs to be some mechanism which is inline with the traditional TV. Furthermore, the huge collection of existing 2D content should be converted to 3D using depth image-based rendering techniques. This conversion technique can highly contribute in overcoming the shortage problem of the 3D content. This paper presents a novel approach for converting 2D degraded image for DIBR 3D-TV view. This degraded or noisy/blur image is enhanced through image dehazing and Directional Filter Bank (DFB). This enhancement is necessary because of the occlusion effect or hole filling problem that occurs due to imperfect depth map. The enhanced image is then segmented into the foreground image and the background image. After the segmentation, the depth map is generated using image profiles. Moreover, Stereoscopic images are finally produced using the DIBR procedure which is based on the 2D input image and the corresponding depth map. We have verified the results of the proposed approach by comparing the results with the existing state-of-the-art techniques.

Partial Text

Depth Image Based Rendering three-dimensional Television (DIBR-3DTV) technology added a new dimension to the world of entertainment. The advancement in DIBR-3D changed the conventional two-dimensional (2D) entertainment media to more realistic one. This diverse technology has adopted by many entertainment platforms such as TV, cinemas and video gaming [1]. The 2D video games have been converted to 3D games by using Kinect camera [1] in XBOX [2], so people can enjoy playing games in a more realistic virtual world representation. Film industries are gaining enormous financial benefits by introducing DIBR-3D technology and their income increased exponentially e.g. ‘avatar’, a DIBR-3D enabled movie released in 2009 and it became the game changer in the film industry which earned tenfold of its investment [3]. Multiple broadcasting service providers use DIBR-3D technology and many European and Asian countries have commenced its transmission. DIBR-3D TV is the revolution in traditional television systems and made it even more smart where viewers enjoy lifelike scenes on their traditional home television [4].

The existing depth map generation algorithms are mainly classified in two categories: Automatic method and Semi-Automatic method. In Automatic method, different depth cues are considered such as focus and defocus information [9, 10] where image’s focus data is considered by varying the focus parameters of a camera. Yang et al. [8] proposed a method which classify the input image into the three categories i.e. landscape, closeup and the linear perspective image. After classification, each class’s depth map is produced. In [12], defocus / blur information is calculated at the edges of objects to approximate the defocus depth map. Huang et al. [11] proposed an algorithm for estimation of depth map which is produced by finding defocus/blur edge information using wavelet transformation and the canny edge detector. Depth from objects motion in video frames has been proposed by [15]. Tsai et al. [16] proposed Gaussian mixture model (GMM) and (SLIC) super pixel simple linear iterative clustering algorithm to generate the initial depth map. The initial depth map further refined using edge’s information and various scanning path mode. In [17], the formation of the depth map is based on Sum of Absolute Difference (SAD) of neighborhood pixels of two same images. Williem et al. [18] proposed anaglyph image based approach to generate the depth map. The obtained depth map assists the algorithm to colorize the synthesis images. The defocus depth map estimation in [19] has achieved in two phases. In the first phase, the defocus/blur image is re-blurred. In the second phase, the ratio of edges’ information is taken of the re-blurred and the input image. Although, automatic methods of depth map generation is less computational expensive, still these methods compromise on the quality of depth map. All the above depth map generation methods are relying on priory defined geometrical information.

A novel approach to generate 3D view is proposed in this paper. Fig 1 is the block diagram of the purposed system. The DFB-DIBR consists of following parts: Image Dehazing, Noisy/Blur Image Enhancement using DFB, grouping background pixels of similar intensity using k-mean, Applying image profiles or depth hypothesis on background pixels, depth map generation and refinement, creating synthesized left and right image using DIBR, creating Anaglyph image or 3D view. It is shown in Fig 1. A blur/noisy image is inserted to the system. The image is dehazed first. Then the dehazed image is enhanced using DBF. After enhancement, the segmented foreground (white) and background (black) image is inserted. The foreground is a bright region and does not contain any depth discontinuity. On the other hand, the background region has depth variation and we are assuming that pixels of similar intensity have similar depth. We have used K-means classification algorithm to group similar intensity pixels and assigned the same depth value at the next stage. Image profile/depth hypothesis is assigned to the classified background which combined with the foreground pixels to generate the initial depth map. The initial depth map is further cultured to retrieve the refined depth map. In next turn, the refined depth map is integrated with the input image to generate synthesized stereoscopic images using DIBR. At the final stage, the synthesized images are combined to generate 3D virtual view. Each part of the system is described in the following parts.

To authenticate the predominance of the DFB-DIBR, numbers of degraded images datasets [37–39] have been tested. In the dataset [37] tested images are Rabbit(800×490), Bear(800×618), Troll(800×563), plant2(800×595), Threads(800×608), Donkey(800×543), Glass(800×532), Plant(800×604) and Plastic bag(800×662). The Data set [37] consists of two types of degraded images. First type of degraded images are photographed by placing the object in front of the monitor seeming natural blur/noisy images. Second type of degraded images are taken in real natural view. The experiments are conducted using a System with Intel Core i7-3632QM CPU(2.20 GHz) having 8GB of RAM. To assess the results of the DFB-DIBR and state of the art methods [12] and [26], the depth map and corresponding anaglyph images have been shown in Figs 10 and 11 respectively. The depth map results of fabricated blur/noisy images, generated by [12] and [26] cannot distinguish the architecture very well. In test sequence “Glass” the texture information of digits written inside the clock is missing in the depth map of [12] and [26]. Whereas such texture information is very much clear in the depth map of the DFB-DIBR due to enhancement of the background data. The relationship between depth values of different intensity pixels are ignored in [12], especially the depth values of different objects are same in the tested image “Threads” which in fact, contain different intensity values and should have assigned different depth values. The proposed system clearly differentiates the high and low-intensity pixels and assigned respective depth values accordingly. The edges information of sky in the test sequence “Rabbit” is missing in the depth map of [12] and [26]. The depth map results of blur/noisy images taken in natural view of the purposed system are far superior than the depth map results of Refs [12, 26]. The depth map of the test sequence “plant2” is properly bedded. In other words, the global depth gradient is maintained by the proposed system whereas such property is ignored by [12] and [26]. To show the performance dominance of the proposed system over conventional algorithms [12] and [26], some full-reference image quality evaluation parameters are used such as Universal image quality index (UQI) [44], Structural Similarity Index (SSIM) [45] and Peak Signal to Noise Ratio (PSNR) [46].

In this paper, we proposed a novel approach to convert a 2D blur/noisy image to 3D view. The image is dehazed first. Then the noisy image is enhanced using DFB. The enhanced image is segmented into background and foreground in the next stage. The foreground is the enhanced part of the image and the background part has intensity variation. The similar intensities are grouped using k-mean algorithm. After grouping similar intensities, image profile/depth hypothesis procedure is applied to generate depth map. The initial depth map is further refined using a bilateral filter to remove some natural artifacts. Moreover, the stereoscopic images are produced using DIBR. Experimental results show the superiority of the proposed novel approach to generate 3D scene from single 2D blur/noisy image. Since the proposed system generates efficient results therefore the future research will focus on using the proposed system as hand crafted feature for the deep learning algorithm.




Leave a Reply

Your email address will not be published.