ECE PhD Dissertation Defense: Sara Banian

PhD Dissertation Defense: Content-Aware AI-Driven Design Assistance Frameworks for Graphic Design Layouts

Sara Banian

Location: Zoom Link

Abstract: Designing user interfaces (UIs) for mobile interaction is widespread but still challenging. It is important for the overall user satisfaction and application success. During the design process, designers express their requirements through images describing the UI’s layout, structure, and content. Designers, however, encounter key challenges throughout the design process. For example, searching for inspiring design examples is challenging because current search systems rely on only text-based queries and do not consider the UI structure and content. Furthermore, these systems often focus on overall page-level layout over individual UI components. Also, creating wireframe templates is difficult for many designers as it necessitates an understanding of different design guidelines. Therefore, it is critical to support designers by developing effective design tools to help them be more productive and creative.
In this dissertation, I aim to explore how to develop design assistance methodologies to augment the process of UI layout design, with a particular focus on visual search and layout generation. Specifically, for this exploration, I seek to investigate the use of advanced deep learning models in the context of mobile UI layout design. Processing layouts differs from processing pixel-level images in that it necessitates processing both the semantic (e.g., labels) and spatial (e.g., coordinates) content of the layout to model the data properly. To achieve this, I explore the design problems from both the data and the model side. First, I present a large-scale UI dataset that accurately specifies the interface’s view hierarchy (i.e., UI components and their location). Second, I contribute the VINS framework, which is composed of three systems LayVis, CompVis, and TransVis that addresses layout-based visual search, component-based visual search, and layout generation, respectively.
First, I introduce LayVis, an object-detection layout-based retrieval model. It takes as input a UI image and retrieves visually similar design examples. Next, I introduce CompVis, a component-based visual search system to easily retrieve individual UI components via convolutional neural networks (CNNs). Specifically, for a given query, the system allows to retrieve (1) text label synonyms, (2) similar UI components, and (3) design examples containing such components. Finally, I present TransVis, a transformer-based generative framework that investigate how to generate UI layouts according to user specifications and following design practices. It specifically models UI layouts as an ordered sequence of elements based on spatial and semantic relationships for (1) generating complete UI layouts, (2) auto-completing existing UI layouts seamlessly, and (3) supporting many design elements per layout.
Overall, the work presented in this dissertation contributes to augmenting the UI layout design. Through quantitative and qualitative evaluation of VINS, we conclude the following: (1) Advanced deep learning models can aid in the development of design assistance methodologies for layout design; and (2) Designers perceive the use of VINS inspiring and useful. Such insights, combined with the open-sourced large-scale dataset, can help the research community develop more effective AI-based data-driven design tools. This work presents future opportunities to investigate different deep learning models within the context of layout design and how designers interact with these AI-based models.