ECE PhD Proposal Review: Sara Banian

PhD Proposal Review: Content-Aware Design Assistance Frameworks for Graphic Design Layouts

Sara Banian

Location: Zoom Link

Abstract: Layout is an important visual communication factor in graphic design that encompasses a page’s overall composition. During the different design stages, designers express their requirements through images describing the interface’s visual layout, hierarchical structure, and content. They create wireframe layouts to meet user requirements and find relative design examples to gain inspiration and explore design alternatives. This is not only an iterative process, but also a time-consuming one.

In this proposal, we aim to design and evaluate design assistance methodologies to augment the process of layout design with a particular focus on visual search and wireframe creation in the context of mobile User Interface (UI) deign. For visual search, we investigate how to find design examples that are relative to the design requirements of a UI layout. Layout retrieval is different from pixel-level image retrieval, as it requires processing both the spatial layout and the content of the data to retrieve similar images. To achieve this, I explore the problem of user interface image retrieval from both the data and the model side, by collecting a more highly annotated, well-suited dataset and proposing an object-detection based image retrieval model. The model takes as input a user interface image and retrieves the visually similar design examples. It uses object detection to identify the user interface components, performs semantic segmentation to produce a hierarchical structure, and trains an attention-aware multi-modal embedding network that leans the structure and content of the given layout design for relevant image retrieval. Results show that the system is capable of retrieving relative design examples through content analysis. Next, I propose a generative framework to investigate how to generate layout wireframes according to user specifications and following common design practices and conventions. The generative framework aims at modeling the content of the UI layouts taking into account different layout variations and design features.