Abstract:
Facial skin analysis is a rapidly evolving field within computer vision and artificial intelli- gence, offering valuable applications in dermatology, skincare, and cosmetic technology. The increasing demand for early skin disease diagnosis and personalized skin health solutions has accelerated the development of intelligent systems capable of identifying facial skin conditions with high precision and providing effective treatment recommenda- tions. This project aims to develop and evaluate a deep learning-based pipeline capable of performing multi-label, multi-class semantic segmentation of facial skin diseases. The primary objective is to identify co-existing skin conditions—such as acne, redness, dark circles, and white spots—with high precision and offer relevant, personalized skincare recommendations. To address this, we propose an end-to-end AI-powered framework titled the Face Disease Detection and Recommendation System, which integrates deep learning, image segmentation, and a Retrieval-Augmented Generation (RAG) pipeline built on a large language model. The proposed system functions in two main phases. In the first phase, facial skin disease detection and segmentation are performed using U-Net, U-Net++, and a custom CNN, trained on three annotated datasets: Acne 04, Dermatology Advisor, and Detection of Skin Diseases. These datasets, formatted in COCO JSON, were sourced from Roboflow and Kaggle. Nine model-dataset combinations were evaluated using accuracy, Dice coefficient, and IoU, with U-Net++ showing the best performance and selected for deployment. To improve facial ROI extraction, a YOLO model trained on a Kaggle Face Detection Dataset isolates the face from user-provided images (front, left, right). These images pass through a two-stage U-Net++ segmentation pipeline—one model trained on the Detection of Skin Diseases dataset and a second, acne-specific model trained on the ACNE-04 dataset—to boost segmentation accuracy for acne. In the second phase, a Retrieval-Augmented Generation (RAG) pipeline using the Gemini Flash 1.5 language model provides personalized skincare product recommendations and answers skin-related queries. This integration of image-based diagnosis with AI-driven recommendations offers users a comprehensive virtual dermatology assistant.
