publications

2024

1. CSCW’24

   Supporting Organizations in Improving Employee Bulk E-mail—A Tool Design and Evaluation Study

   Kong, Ruoyan, Sun, Ruixuan, Zhang, Charles Chuankai, Yuan, Ye, and Konstan, Joseph A

   Proceedings of the ACM on Human-Computer Interaction 2024

   Abs HTML

   Organizations often send bulk emails to employees to make them aware of policy changes, organization plans, and events. Many of these emails, however, are long digests with many separate messages that waste employees’ time and reduce their awareness. This study introduces CommTool–a prototype tool to help organizational communicators better understand their emails’ performance and cost. We first interviewed 5 communicators and identified the need to measure the performance of each message within bulk email. Then we iteratively designed and deployed an organizational bulk email evaluation platform (CommTool), which enables communicators to get diverse message-level metrics such as reading time, relevance rate, comments, etc. We evaluated these designs through a 2-month field deployment with 5 communicators and 149 organization employees. We found that 1) the message-level metrics, such as reading time and relevance rate, helped communicators understand their audience and design bulk emails; 2) the cost and reputation metrics did not influence the organization leaders’ decisions. We summarize with suggestions on designing organizational bulk email evaluation platforms that provide message-level performance and cost information.

2. RecSys’24

   AI-based Human-Centered Recommender Systems: Empirical Experiments and Research Infrastructure

   Sun, Ruixuan

   In Proceedings of the 18th ACM Conference on Recommender Systems 2024

   Abs HTML

   This is a dissertation plan built around human-centered empirical experiments evaluating recommender systems (RecSys). We see this as an important research theme since many AI-based RecSys algorithmic studies lack real human assessment. Therefore, we do not know how they work in the wild that only human experiments can tell us. We split this extended abstract into two parts – 1) A series of individual studies focusing on open questions about different human values or recommendation algorithms. Our completed works include user control over content diversity, user appreciation on DL-RecSys algorithms, and human-LLMRec interaction study. We also propose three future works to understand news recommendation depolarization, personalized news podcast, and interactive user representation; 2) An experimentation infrastructure named POPROX. As a personalized news recommendation platform, it aims to support the longitudinal study needs from the general AI and RecSys research community.

3. WikiWorkshop’24

   Investigating Social Interaction Factors for Newcomer Retention in Wikipedia Teahouse

   Zhou, Moyan, Sun, Ruixuan, and Terveen, Loren

   2024

   Abs HTML

   Newcomers benefit online communities: they bring innovative and diverse ideas and fill in the gaps created by members who leave the community (Morgan and Halfaker, 2018). However, onboarding newcomers and retaining them is challenging (Halfaker et al., 2013), as newcomers tend to drop out even after their first session (Karumur et al., 2016). Thus, online communities dedicate efforts to understand and to promote newcomer retention, one approach being socialization. For example, Wikipedia Teahouse, a Q&A forum, encourages social interactions between newcomers and more experienced editors. Prior work has found that the Teahouse supports newcomer retention (Morgan et al., 2013). Loren Terveen University of Minnesota Minneapolis, Minnesota, USA whoaskquestions are referred to as “guests,” while more experiencededitorswhoanswerthequestionsarereferred to as “hosts.” We follow this convention. And we hypothesize variables based on prior work in knowledge sharing communities, member motivation and participation. We divide these variables into two broad groups, shown in Table 1. The positive effect of Wikipedia Teahouse hints relationships between socialization and newcomer retention, which is essential to the success of online communities. Thus, to further investigate this relationship, we ask: how do social interactions with other community members affect newcomer commitment in peer production platforms? In the context of Wikipedia Teahouse, we ask: how does Teahouse interactions affect newcomer retention? Answering this question will provide insights for communitymembersonhowtointeractwithnewcomers, potentially extending to other forms of social interactions such as talk page discussions. In this workshop paper, we consider Wikipedia Teahouse as a case study, and unpack how social interactions with more experienced community members affect newcomers’ commitment to Wikipedia. We aim to understand and thus enhance socio-technical interventions designed to retain newcomers by socialization, and to assist newcomer decline in the long term. By identifying specific factors that correlate with newcomers’ retention inEnglishWikipedia,wecontributetoWikipediathrough the following research questions: RQ: What and how do social interaction factors affect newcomer retention in Wikipedia?

4. IJHCI

   Interactive content diversity and user exploration in online movie recommenders: A field experiment

   Sun, Ruixuan, Akella, Avinash, Kong, Ruoyan, Zhou, Moyan, and Konstan, Joseph A

   International Journal of Human–Computer Interaction 2024

   Abs HTML

   Recommender systems often struggle to strike a balance between matching users’ tastes and providing unexpected recommendations. When recommendations are too narrow and fail to cover the full range of users’ preferences, the system is perceived as useless. Conversely, when the system suggests too many items that users don’t like, it is considered impersonal or ineffective. To better understand user sentiment about the breadth of recommendations given by a movie recommender, we conducted interviews and surveys and found out that many users considered narrow recommendations to be useful, while a smaller number explicitly wanted greater breadth. Additionally, we designed and ran an online field experiment with a larger user group, evaluating two new interfaces designed to provide users with greater access to broader recommendations. We looked at user preferences and behavior for two groups of users: those with higher initial movie diversity and those with lower diversity. Among our findings, we discovered that different levels of exploration control and users’ subjective preferences on interfaces are more predictive of their satisfaction with the recommender.

2023

1. UMAP ’23

   Less Can Be More: Exploring Population Rating Dispositions with Partitioned Models in Recommender Systems

   Sun, Ruixuan, Kong, Ruoyan, Jin, Qiao, and Konstan, Joseph

   In Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization 2023

   Abs HTML

   In this study, we partition users by rating disposition - looking first at their percentage of negative ratings, and then at the general use of the rating scale. We hypothesize that users with different rating dispositions may use the recommender system differently and therefore the agreement with their past ratings may be less predictive of the future agreement. We use data from a large movie rating website to explore whether users should be grouped by disposition, focusing on identifying their various rating distributions that may hurt recommender effectiveness. We find that such partitioning not only improves computational efficiency but also improves top-k performance and predictive accuracy. Though such effects are largest for the user-based KNN CF, smaller for item-based KNN CF, and smallest for latent factor algorithms such as SVD.

2. ETRA ’23

   Getting the Most from Eye-Tracking: User-Interaction Based Reading Region Estimation Dataset and Models

   Kong, Ruoyan, Sun, Ruixuan, Zhang, Charles Chuankai, Chen, Chen, Patri, Sneha, Gajjela, Gayathri, and Konstan, Joseph A.

   In Proceedings of the 2023 Symposium on Eye Tracking Research and Applications 2023

   Abs HTML

   A single digital newsletter usually contains many messages (regions). Users’ reading time spent on, and read level (skip/skim/read-in-detail) of each message is important for platforms to understand their users’ interests, personalize their contents, and make recommendations. Based on accurate but expensive-to-collect eyetracker-recorded data, we built models that predict per-region reading time based on easy-to-collect Javascript browser tracking data. With eye-tracking, we collected 200k ground-truth datapoints on participants reading news on browsers. Then we trained machine learning and deep learning models to predict message-level reading time based on user interactions like mouse position, scrolling, and clicking. We reached 27% percentage error in reading time estimation with a two-tower neural network based on user interactions only, against the eye-tracking ground truth data, while the heuristic baselines have around 46% percentage error. We also discovered the benefits of replacing per-session models with per-timestamp models, and adding user pattern features. We concluded with suggestions on developing message-level reading estimation techniques based on available data.

3. CHI ’23

   Collaborative Online Learning with VR Video: Roles of Collaborative Tools and Shared Video Control

   Jin, Qiao, Liu, Yu, Sun, Ruixuan, Chen, Chen, Zhou, Puqi, Han, Bo, Qian, Feng, and Yarosh, Svetlana

   In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems 2023

   Abs HTML

   Virtual Reality (VR) has a noteworthy educational potential by providing immersive and collaborative environments. As an alternative but cost-effective way of delivering realistic environments in VR, using 360-degree videos in immersive VR (VR videos) received more attention. Although many studies reported positive learning experiences with VR videos, little is known about how collaborative learning performs on VR video viewing systems. In this study, we implemented two collaborative VR video viewing modes based on the way of group video control, synchronized or shared (Sync mode) and non-synchronized or individual (Non-sync mode) video control, against a conventional VR video viewing setting (Basic mode). We conducted a within-subject study (N = 54) in a lab-simulated remote learning environment. Our results show that collaborative VR video modes (Sync and Non-sync mode) improve users’ learning experiences and collaboration quality, especially with shared video control. Our findings provide directions for designing and employing collaborative VR video tools in online learning environments.

4. ICWSM ’23

   We Are in This Together: Quantifying Community Subjective Wellbeing and Resilience

   Dong, MeiXing, Sun, Ruixuan, Biester, Laura, and Mihalcea, Rada

   In Proceedings of the International AAAI Conference on Web and Social Media 2023

   Abs HTML

   The COVID-19 pandemic disrupted everyone’s life across the world. In this work, we characterize the subjective wellbeing patterns of 112 cities across the United States during the pandemic prior to vaccine availability, as exhibited in subreddits corresponding to the cities. We quantify subjective wellbeing using positive and negative affect. We then measure the pandemic’s impact by comparing a community’s observed wellbeing with its expected wellbeing, as forecasted by time series models derived from prior to the pandemic. We show that general community traits reflected in language can be predictive of community resilience. We predict how the pandemic would impact the wellbeing of each community based on linguistic and interaction features from normal times before the pandemic. We find that communities with interaction characteristics corresponding to more closely connected users and higher engagement were less likely to be significantly impacted. Notably, we find that communities that talked more about social ties normally experienced in-person, such as friends, family, and affiliations, were actually more likely to be impacted. Additionally, we use the same features to also predict how quickly each community would recover after the initial onset of the pandemic. We similarly find that communities that talked more about family, affiliations, and identifying as part of a group had a slower recovery.

2022

1. CSCW ’22

   Multi-Objective Personalization in Multi-Stakeholder Organizational Bulk E-Mail: A Field Experiment

   Kong, Ruoyan, Zhang, Charles Chuankai, Sun, Ruixuan, Chhabra, Vishnu, Nadimpalli, Tanushsrisai, and Konstan, Joseph A.

   In Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2), pp.1-27. 2022

   Abs HTML

   Bulk email is often used in organizations to communicate "important-to-organization” messages such as policy changes, organizational plans, and administrative updates. However, normal employees may prefer messages more relevant to their jobs or interests. Organizations face the challenge of balancing prioritizing the messages they prefer employees to know (tactical goals) while maintaining employees’ positive experiences with these bulk emails, then they continue to read these emails in the future (strategic goals).Could personalization help organizations achieve these tactical and strategic goals? In an 8-week field experiment with a university newsletter, we implemented a 4x5x5 factorial design on personalizing subject lines, top news, and message order based on both the employees’ and the organization’s preferences. We measured these designs’ influences on the open/interest/recognition/read-in-detail rate of the whole newsletter and the single messages within it.We found that ”important-to-organization” messages only got higher recognition rates when being put on subject lines / top news (tactical goal). Mixing them with employee-preferred messages in top news did not bring further improvement to their own recognition rates but could improve the whole newsletter’s recognition rate. Only when the top news solely contained the employee-preferred messages were the employees slightly more interested in the newsletter (strategic goal). We further analyze on which topics the employees and the organization’s preferences conflicted. Finally, we discuss the design suggestions for organizational bulk email.

2. IEEE TSMC

   Neural Reranking-Based Collaborative Filtering by Leveraging Listwise Relative Ranking Information

   Li, Fan, Qu, Hong, Fu, Mingsheng, Zhang, Liyan, Zhang, Fan, Chen, Wenyu, Sun, Ruixuan, and Zhang, Haixian

   IEEE Transactions on Systems, Man, and Cybernetics: Systems 2022

   Abs HTML

   Reranking is a critical task used to refine the initial collaborative filtering (CF) recommendation by incorporating information from different viewpoints, such as the extra item side-information and user profile. In this article, a neural reranking-based CF (NRCF) model is proposed to leverage composite viewpoints from the basic CF model and user preference. More precisely, the predictive implicit user preference is first constructed from the initial top- k items. The implicit user preference is then aggregated with the explicit user embedding to enrich the user intent representation. Moreover, the traditional listwise loss functions for reranking optimization are suboptimal, due to the fact that they neglect the relative ranking information (ReinRank) between the unobserved and positive items. To address this issue, a novel listwise loss function that leverages relative ranking information, referred to as ReinRank, is proposed for reranking optimization. ReinRank assigns different values to the unobserved items, according to their relative ranking distances between the positive items. Extensive experiments are performed on three public benchmarks and different CF models, in order to demonstrate the effectiveness of NRCF and ReinRank.

2021

1. ICAART ’21

   Twin-GAN for Neural Machine Translation

   Zhao, J., Huang, L., Sun, R., Bing, L., and Qu, H.

   ICAART 2021

   Abs HTML

   In recent years, Neural Machine Translation (NMT) has achieved great success, but we can not ignore two important problems. One is the exposure bias caused by the different strategies between training and inference, and the other is that the NMT model generates the best candidate word for the current step yet a bad element of the whole sentence. The popular methods to solve these two problems are Schedule Sampling and Generative Adversarial Networks (GANs) respectively, and both achieved some success. In this paper, we proposed a more precise approach called “similarity selection” combining a new GAN structure called twinGAN to solve the above two problems. There are two generators and two discriminators in the twin-GAN. One generator uses the “similarity selection” and the other one uses the same way as inference (simulate the inference process). One discriminator guides generators at the sentence level, and the other discriminator forces these two generators to have similar distributions. Moreover, we performed a lot of experiments on the IWSLT 2014 German→English (De→En) and the WMT’17 Chinese→English (Zh→En) and the result shows that we improved the performance compared to some other strong baseline models which based on recurrent architecture.

2019

1. GT Symposium

   An Evaluation of the Value of Lip and Jaw Motion in Virtual Reality Avatars

   Sun, Ruixuan

   2019

   Abs HTML

   Virtual reality (VR) is known as a simulated 3D immersive technology. Currently, the emphasis has been mainly placed on tracking the upper face, like eye tracking and eyebrow imitation, where the lower face containing the largest emotions of the face are usually hard for commercial VR headset to capture due to hardware limit. In this study, we explore the role of lip and jaw motionswhen VR is used as a communication medium by comparing the effectiveness of camera-based facial landmark tracking against audio-driven lip movements.