Selfie cameras are an essential part of any smartphone — but finding the sweet spot for the front camera can often be a barrier to a smooth viewing experience. However, thanks to the Under Display Camera (UDC), users can say goodbye to the little black dot on the front of their screens. Samsung Electronics has given SVP a serious upgrade in the new Galaxy Z Fold4, giving users an even more immersive and uninterrupted full-screen experience.
Like so many other great innovations, the latest UDC was a group effort — guided by Samsung experts and valuable consumer feedback. To learn more about the improved SDU for Galaxy Z Fold4, Samsung Newsroom spoke with experts from Samsung Research,1 who collaborated on the technology.
Read on to see what John Seok-Jun Lee, Head of Computational Camera Section at Samsung Research America (SRA), and Alok Shukla, Head of AI Vision Solutions Section at Samsung R&D Institute India-Bangalore (SRI-B ), had to say about their collaboration behind bringing improved UDC technology.
Q: What is an Under-Display Camera (UDC)?
Seok Jun: An under-screen camera (UDC) is an innovative camera solution that enables an uninterrupted viewing experience on a mobile device by hiding the camera under the screen and dedicating the entire screen to users while applications are running. execution. This requires not only hardware innovation by placing a camera under a display panel, but also algorithmic innovation to restore image quality. — one of the most complex image restoration problems.
Q: How does UDC work?
OK : In UDC, a front camera sits below the screen. The screen’s pixel density is reduced in this area, allowing the screen to display images unhindered — but at the same time, do not block the light from reaching the camera when in use. Because the camera is placed below the display, the UDC can suffer from poor image quality caused by diffraction artifacts, resulting in flare, saturation, blurring, and haze. Therefore, although UDC brings a better display experience, it also affects the image quality of the camera and other downstream vision tasks. These complex and diverse distortions make restoring UDC images extremely difficult.
Q: Why is UDC an important technology when it comes to the Galaxy Z Fold4? What value does it bring to customers?
Seok Jun: Samsung’s foldables are designed to be new and different, combining smartphone and immersive viewing experiences in one mobile device through revolutionary display technology. When folded, it provides smartphone-like usability, and when unfolded, the screen provides a more immersive and productive experience. Our consumer research showed that Galaxy Z Fold series users prefer the Cover Screen Camera for capturing photos and selfies and the immersive Main Display for more productive tasks like watching videos, playing games, and working. on documents. Therefore, we felt that UDC technology would be the ideal solution to meet their need for an uninterrupted screen while still being able to perform tasks such as opening video calls. Samsung launched the first-ever foldable UDC solution in the Galaxy Z Fold3 and further improved its quality in the Galaxy Z Fold4.
OK : The Galaxy Z Fold4 truly unveils an immersive display world. UDC technology is at the heart of the quest for a true full-screen display. By combining cutting-edge innovation in display technology with advanced AI techniques for image restoration, the Galaxy Z Fold4 strikes the perfect balance between screen quality and image quality. of the camera.
Q: How has the camera image quality improved since the Galaxy Z Fold3?
OK : Samsung Galaxy Z Fold3 introduced the world’s first UDC on a foldable. After the Galaxy Z Fold3 was released, one of the main consumer feedbacks was to make SVP less visible. On the Galaxy Z Fold4, the sub-pixel layout has been optimized to further hide the UDC.
As mentioned earlier, when the camera is placed below the screen, it can lead to poor image quality caused by diffraction artifacts, which cause flare, saturation, blurring and haze.
That’s why, in Galaxy Z Fold4, we’ve harnessed the power of AI, backed by multi-image processing, to develop a new computational imaging pipeline for UDC image restoration.
SRI-B, with Visual SW R&D Group at MX Business, focused on the development of AI image restoration for photo capture and preview to compensate for optical diffraction in UDC. At the same time, SRA’s Mobile Processor Innovation (MPI) lab improved the multi-frame computational photography algorithm. The Galaxy Z Fold4 UDC solves the main technical challenges in image quality to deliver a unique experience to end customers.
Q: Tell us about the journey of the SRA-SRI partnership project. Why did your two teams collaborate?
Seok Jun: The SRA MPI Lab has worked with SRI-B throughout MPI’s history at Samsung, from the release of the Galaxy Note4 camera in 2014 to the Galaxy Z Fold4 and Z Flip4 in 2022. SRA MPI led research and development of MX multi-frame processing (MFP) technology for UDC, Night Mode, Expert RAW and other technologies. SRI-B has contributed to all released SRA MPI MFPs by providing amazing camera software, image quality tuning and resources for commercialization as well as R&D on AI cameras. Our 3-way collaboration with the MX R&D group, SRA MPI and SRI-B has been a success — especially when the global team made effective use of time zone differences to complete the milestones of such a complicated project on time. It’s always a pleasure to work with them and bring the camera successes of Galaxy devices together to achieve our common goal of being the best smartphone camera experience in the world.
Q: Please explain your role and your team’s contributions to UDC. How did you help make the best user experience possible?
OK : Samsung Galaxy Z Fold4 UDC was a complex R&D project, and SRI-B’s role was to think end-to-end for the best consumer experience. This required multi-domain expertise and deep collaboration between multiple teams. Experts from SRI-B, SRA-MPI Lab, and MX Visual SW R&D Group worked seamlessly to deliver an end-to-end UDC experiment.
SRI-B was also key in using a Neural Processing Unit (NPU) to run the AI model, achieving real-time performance in previews and significantly faster capture processing time. than in the Samsung Galaxy Z Fold3 UDC. Camera previews, video recordings, and video call scenarios require real-time performance without dropping any frames. Accelerated AI-based UDC restore with NPU for preview processing was first applied in Samsung Galaxy Z Fold4. Although real-time processing was not necessary, since camera capture requires higher image quality than previewing, it was desirable to achieve better image quality with less processing time for user experience.
Q: What were the challenges you faced while developing this technology, and how did you overcome them?
Seok Jun: Low light transmission and light diffraction are the main challenges of UDC camera technology. Yet another challenge, which is different from a punch-hole camera image pipeline, is that the UDC capture quality is affected not only by the camera module, but also by the display panel. SRA MPI’s AI-based 3D ISP algorithm has been designed with the flexibility to adjust the output image quality with different display characteristics. Having such flexibility in 3D ISP helped overcome these challenges.
OK : The main objective of SRI-B was to develop an AI image restoration module for Samsung Galaxy Z Fold4 UDC. These restorations are necessary for the photo capture mode as well as the video mode. Both video recording and video call scenarios have strict requirements for real-time processing within current consumption limits.
AI-based UDC restore was first applied in Samsung Galaxy Z Fold4 UDC for video. To get better video quality than the Galaxy Z Fold3, we had to lighten the AI model while still getting good restore quality. Several experiments have been conducted in the design and training of networks to achieve good real-time quality. SRI-B made extensive use of NPU and advanced quantization techniques to prepare a lighter and highly optimized AI model that achieves real-time performance with low current consumption.
Q: What is your vision for future UDC iterations at Samsung?
OK : Samsung Galaxy Z Fold4’s UDC has opened up a new dimension to the smartphone camera form factor. UDCs in the future will allow larger camera sensors and multiple cameras to be placed at flexible positions without compromising the viewing experience. Another big challenge is the massive adoption of UDC technology in more products as the main front camera. For this to happen, it will take a significant improvement in camera image quality, processing time, and display experience.
Seok Jun: Since the Galaxy Z Fold4 UDC offers a much better experience in terms of hiding the camera under the display and improving image quality, the direction of R&D for upcoming flagship devices is changing. will endeavor to make the UDC completely invisible. This will provide the best uninterrupted immersive mobile display experience while achieving the same image quality as a punch-hole camera.
1 Samsung Research, acting as Samsung Electronics’ advanced R&D center, leads the development of future technologies for the company’s Device eXperience (DX) division.