IConCMT 2023

Scalable Interactive Visual Analysis Through Storytelling 

Eduard Gröller - computer scientist and professor for visual computing and human-centred technology

Scalable Interactive Visual Analysis combines computer-supported, interactive, visual representations of (abstract) data with automatic techniques to amplify cognition and facilitate modeling. In recent years data complexity has grown considerably, and data-driven storytelling has gained increased interest. To exemplify data-driven storytelling, we present molecumentaries for producing documentary-style content using real-time scientific visualization. Molecular documentaries feature structural models from molecular biology, created through adaptable methods instead of the rigid traditional production pipeline.

Our work is motivated by the rapid evolution of scientific visualization and its potential in science dissemination. Without some form of explanation or guidance, however, novices and lay-persons often find it difficult to gain insights from the visualization itself. We integrate such knowledge using the verbal channel and provide it along an engaging visual presentation. We turn texts written by domain experts into verbal representations using text-to-speech functionality and provide them as a commentary. Afterwards storytelling as promising communication medium will be put into a social context. Possible future directions will be briefly touched in areas like immersive/situated analytics, conversions between text and visuals, and industry 4.0 to 6.0.

Elvin Luku: Recreating News on TikTok: Challenges and Opportunities for Mainstream Media to Reach New Audiences

The audiences of traditional media are getting older, while the young generations have chosen social media as the main source of information and entertainment. The study of audience behavior from the Department of Journalism & Communication in the University of Tirana have found that Generation Z (under 25s) “spent more than 4 hours using a smartphone, but none of them to consume news during this time”. Meanwhile the latest survey of MediaLook Center (2020) has identified that “Instagram and TikTok are the main source of information for more than 70% of teenagers”.  These trends comply with the latest international report of Reuters Institute (2022) which underlines that "the so-called GenZ have embraced Instagram and TikTok for news, entertainment and distraction, to express their political rage – but also to tell their own stories in their own way”.   
Taking in consideration these data and the challenge of the traditional newsrooms to reach the young audiences this paper raises several research questions: Can traditional media produce reports, storytelling and criticism through Tik Tok’s specific language and visual rhetoric in the platform to gain the attention of GenZ? May these new forms of video production be considered as a journalism genre in the social media 3.0? Can traditional newsrooms have a different approach of various social, political, economic and cultural phenomena, to produce infotainment Tik Tok videos? 
This study employs a participatory research methodology. The author of this paper has been part of the board of ethics of the first Tik Tok newsroom in Albania called TikTok Corruption. During a seven-month period from May 1 to November 30, 2020 the team of 5 Tik Tokers have published 366 productions on the platform, based on the social, political and economic developments in Albania. This productions have been analyzed using a quantitative and qualitative method, aimed at highlighting the limitations and achievements of this new form of journalism on Tik Tok. Moreover, five interviews have been conducted with young Tik Tokers. The preliminary findings indicate that people with a journalistic background have found difficulties to adapt to the Tik Tok culture of storytelling, where the journalist conveys information through his own performance. Secondly, Tik Tokers face difficulties to express through visual symbolism and therefore they complement visual rhetoric with long explanatory texts. 

Sohail Dahdal: Enhancing Youth Engagement in Climate Action Messages through Virtual Reality Immersive Gameplay: A Case Study

Effective communication of critical messages, such as the urgency of addressing climate change, to the youth demographic poses a formidable challenge. Often, the call to action is subtly embedded within the message, limiting the degree of engagement required to catalyze meaningful responses (Bandura, Cherry, 2020). This paper investigates this issue by presenting a case study centered on a virtual reality (VR) immersive series game meticulously crafted to convey the intricacies of climate change. Unlike conventional approaches, this VR game merges narrative-driven gameplay with a embedded call to action, thereby enabling participants to comprehend the issue, explore potential solutions, and pledge proactive measures (Boncu, Candel, Popa, 2022). 
Virtual Reality, recognized for its capacity to induce a heightened sense of presence and immersion (Sanchez-Vives, Slater, 2005), offers a unique platform for amplifying engagement. By enveloping participants within the narrative, VR holds the potential to elevate the youth's involvement in serious matters. The integration of gameplay elements further enhances this potential by fostering a deeper connection with the subject matter (Laamarti, Saddik, 2014). This augmented engagement not only provides a more effective means of conveying complex concepts but also contributes to increased motivation for real-world action beyond the virtual realm (Hügel, Davies, 2022). 
Through the lens of the case study, this paper demonstrate the effectiveness of the VR immersive serious games in bridging the gap between message delivery and tangible action (Martini, Resceanu, Tilea, 2023). It explores how the combination of immersive storytelling and interactive gameplay influences youth perception, knowledge retention, and willingness to commit to climate-conscious behaviors. The findings suggest that the synergy between VR immersion and gameplay mechanics is conducive to fostering a sense of personal responsibility and agency, leading to a higher likelihood of meaningful actions being taken outside the confines of the game environment. 
In conclusion, this paper underscores the potential of VR immersive gameplay as a transformative tool for conveying critical messages to the youth. By facilitating a profound understanding of intricate issues like climate change and nurturing a stronger connection to these matters, VR has the capacity to propel the younger generation towards more active and impactful participation in addressing global challenges. This research contributes valuable insights into the realm of innovative persuasive communication strategies, where technology converges with meaningful engagement to inspire genuine change. 

Aline Roes Dalmolin, Ada Cristina Machado Silveira and Camila Hartmann: Parallel Brazil: Analysis of the Bolsonarist Media Sphere

After the second round of the Brazilian presidential elections in 2022, a large part of the population denied basic principles of the democratic process. Supporters of former President Jair Bolsonaro contested the result of the polls, refusing to accept the victory of Luis Inácio Lula da Silva. Reflecting on the communicational and media processes that allow these episodes to occur, is essential to understand the consequences of Bolsonarism and the many ills it creates or exposes. It is a refusal of communicational barbarism, seeking elements to combat it based on the understanding of the circuits and circulatory processes typical of the far-right media anatomy. 
Brazil has an environment favourable to the dissemination of disinformation and with freedom of expression problems, as well as lack of public trust, encourage the consecration of autocratic societies, or the confrontation between visions of exercise of political power (Stier, 2015). This is accompanied by a journalism institutional crisis scenario, which reflects on the constant decrease in their circulation (IVCB, online) provoked by decreased renown print vehicles in Brazil business model. In addition, the productive structures are still very much linked to industrial logic, there is frequent contestation of the credibility of the national mainstream media. Readers migrate to other information sources taken as more credible and with innovative publication formats, such as WhatsApp and Telegram groups’ contents. 
Our proposal is to investigate the Bolsonarist media sphere (Rocha, 2023). That media sphere aims to produce fake news and polarizing narratives (good against evil, left against right), creating a parallel reality and uniting political militancy. In this context, communication is restricted to the exercise of the phatic function within polarized bubbles. The extreme right polarization structure exhausts the communication process, promoting discirculation. The intense circulation in an environment of communicational barbarism, paradoxically, configures itself in discirculation and the informational moral panic (Carlson, 2018) is correlated with the emergence of that cognitive dissonance process. 
This text analyzes Bolsonarist media sphere exemples, more specifically two Bolsonarist channels on Telegram between the 1st and the 8th of January 2023, in which an intense amount of posts and disinformation content circulated daily. The chosen clip covers the period of one week between the inauguration ceremony of Luís Inácio Lula da Silva as president of Brazil, which took place on January 1st, and the January 8th riots, with the invasion and depredation of the National Congress, the Federal Supreme Court (STF) and the Planalto Palace by radical Bolsonarists in brazilian capital. The Bolsonarist Telegram groups were observed from the methodological perspective of mediatized interactional circuits (Braga, 2017). It is expected to observe an escalation of radicalized mobilization, in which the members of the group were summoned to participate in anti-democratic acts. On the other hand, the mobilization of these groups are being investigated by Brazilian Judiciary and police agencies, in order to determine the responsibilities for anti-democratic acts and seek to punish the participants and those responsible. 

(Chair: Yulia Belinskaya)

Video Presentation and remote talk

Thomas Weibel: Mont - An Open Data Mountain Peak Recognition App in Vanilla JS

Gerhard Sprung and Martin Schweighofer: Project DesART: Utilising an AR Sand Table to Understand Complex Interconnections

Julian Rubisch and Tobias Leibetseder: Vibrate resonAlte - Prompt the Artist

Robert Halwaß, Steffen Prowe, Evelyne Becker, Joachim Villwock, Martina Mauch, Linnea Pehl, Clara Simon, and Lena Ziesmann: Immersive STEM -Laboratories to Improve Teaching and Learning (Demo)

Alexander Rind, Wolfgang Aigner, Magdalena Boucher, Florian Grassinger, Stefanie Größbacher, Lydia Popp, and Johannes Pflegerl: Data-Driven Infographics for Counselling Interviews with Family Caregivers in Dementia Care

Dementia is a cognitive disorder that progressively impairs activities of daily living. Thus, a high level of care is needed, which is very often provided by family members or other informal caregivers. This puts family caregivers under a lot of physical and emotional strain.  The Angehörigendialog is a social-diagnostic tool that was developed to identify and alleviate the needs of family caregivers in the Austrian region Burgenland. It provides a questionnaire that mobile nurses use as guide for counseling interviews with family caregivers. Then, a visual report of the individual answers is used to reflect the caregiver’s situation reflected together so that coping strategies and available support can be discussed. 
This talk presents a follow-up project, in which we designed and implemented a web-based questionnaire and bespoke data-driven infographics for the report. The tool was created in a human-centered design process in cooperation between the visualization research group of and institute of social inclusion research. Target users were involved in focus group interviews in the beginning and finalization of the project. The resulting infographics designs are, on the one hand, robust regarding the possible inputs that care givers can provide in the questionnaire. On the other hand, they apply metaphors to be easily understandable and motivate a resource-oriented reflection. The software and the designs are available as free and open-source software at https://github.com/fhstp/angehoerigendialog and can be tested at https://angehoerigendialog.fhstp.ac.at/.  The Angehörigendialog is a case study for a visualization scenario that involves data at a tiny, personal scale that is visualized for personal reflection and as support mutual communication in the counseling interviews. Thus, it marks the intersection of the fields of collaborative data visualization with personal visualization.
 

Julia Boeck, Alexander Rind and Florian Zahorka: Personnel Fluctuation in Austrian Emergency Medical Services: A Data Visualisation Approach

Ensuring steady medical care in Austria's Emergency Medical Services depends heavily on paramedics. Experts in this field stress the importance of understanding paramedic turnover in a systematic way to maintain consistent medical care. Various factors contribute to paramedics leaving their stations, making it challenging for EMS stations to prevent dropouts. To address this, an annual meeting between paramedics and team managers aims to tackle exit reasons, but some factors may go unnoticed. To assist commanders and team managers, a visualization method was developed to track paramedic data. This includes identifying paramedics not meeting shift requirements and showcasing their preferred shift partners and types. Developed through a design study methodology and interviews with two ambulance station experts, the visualizations—mosaic-like, lollipop chart, slope graph, and highlighted table—proved beneficial. They enable commanders to grasp turnover patterns, helping to find solutions. In essence, these visualizations are valuable according to rural ambulance station domain experts. 

(Chair: Kajetan Enge)

„Raise the Treasure!“ - Cultural Data and its potential for visualization & storytelling

Kasra Seirafi - Co-founder of Fluxguide

The lecture highlights current possibilities, trends and technologies for cultural mediation. Mobile computing, AR/VR/XR, data analytics, visualization methodologies, and more, offer new opportunities for outreach, learning and digital innovation to museums. We will connect an EU funded research project („InTaVia“) with museum/cultural practice (Web Application for the „Stiftung Preussische Schlösser und Gärten Berlin-Brandenburg“), and discuss concepts, technologies and lessons learned. We will show approaches how to raise the cultural treasure of exhibitions, collections, and cultural data and how to make it available for experts, the broad public and education.
 

Bruno Caldas Vianna: Visual Artificial Intelligence's Matter of Taste

This paper delves into the concept of taste as a classifier and its role in perpetuating cultural capital. Pierre Bourdieu's study on class and taste highlights how those with higher cultural capital dictate societal notions of good taste, influencing dominated classes. In the era of machine learning visual generative methods, the perpetuation of cultural capital ownership occurs through statistical processes. 

The evaluation of visual neural networks' aesthetic quality involves two crucial steps: filtering out low-quality images and creating test synthetic images for evaluation during training. However, these evaluations are biased, reflecting the preferences of a select group of individuals. Aesthetic evaluation data is obtained through public rating systems, but most users belong to a specific demographic, leading to further homogeneity in taste. Consequently, current rating systems reinforce a limited cultural capital rooted in access to technology and computational creativity. 

To foster diversity in neural generative systems, the paper proposes new scoring systems, incorporating ratings from individuals in the Global South, those unfamiliar with generative systems or computers, and marginalized cultures. This endeavor aims to build a more inclusive aesthetic guidance and address the dominance of specific cultural capital in the field of visual culture. 

Alaz Okudan: Operational Mishap and Unexpected Outcomes: Failed Imagery and Accidental Aesthetics of Artificial Intelligence

This proposal offers an investigation into the user reception of accidental aesthetics encountered in AI-generated images produced by text-to-image technology. Although it is a technology that draws growing interest, AI’s technical and social norms, as well as its dialogue with its users, are still in the developmental stages. Furthermore, due to the discrepancy and incongruity between language and the visual world, AI-generated image platforms carry a strong capacity to create results that may transcend user expectations. With the hypothesis that the appeal of AI-generated images partly stems from their ability to produce unexpected and accidental outcomes that can challenge human expectations, this research aims to comprehend how accidents and failures are perceived in AI images by their users and how their interaction with such results affect their artistic expression and critical thinking. 

Isabel Pina: Media Arts Suspensions and Deviations: Paz Dos Reis between Cinema and Photography

Media have non-linear evolutions. This means that new media failures or retractions can highlight overlooked features of old media. The very early history of Portuguese Cinema is one of those moments: the new and promising cinema technologies were first experimented and then discarded by Aurélio da Paz dos Reis, the pioneer of Portuguese cinema. Instead, he intensely explored in photography a solution for covering current affairs. 
The creation of the ‘Portuguese Stereoscope’, a commercial stereoscopic card series that depicted political events such as the agitated republic rallies, the social life of the bourgeoisie and the daily life in the streets of Porto, was an international innovation: it consisted of the very first photojournalism in 3D. 
This research project will investigate this historical fact by checking in the press the events covered by Paz dos Reis and by creating an online media art installation that will allow the matching of these materials with his stereos to understand how these 3D news materials relate to current VR 360º new videos. 
This digital curation will be developed as an artistic research method which will also allow to experiment with film materials: Aurelio’s original footage, a new short film shot with his original camera kept by the Portuguese Cinematheque (ANIM) and a documentary film focused on some of his most symbolic photographs. The short film we propose to be shot with his original camera is part of an experimental media archaeology method that will be a core part of this research. It is expected that this research project leads to a new approach towards the work of Paz dos Reis. 

(Chair: Markus Wintersberger)
 

Bärbel Bissinger, Christian Märtin and Christian Herdin: Applied Emotion AI in Video Conferences

Artificial Emotional Intelligence, Affective Computing or Emotion AI deals with the ability of machines to recognize human emotions. Our physical signals can be analyzed and categorized which makes it possible to train machines to recognize emotions and respond to it. This changes how we interact with technology, and it could also change how we interact with each other. There are more and more research activities in this field as well as products on the market that apply Emotion AI. According to a recent forecast, emotion detection and recognition is a rapidly growing market which will be worth more than 42 billion USD by 2027. As technology becomes ubiquitous in interpersonal interactions and activities, Emotion AI could make our tool-based interactions more human-like. 
Emotions play a central role in our communication and decision-making and should therefore get more attention, even in business environments. Since Covid-19, more and more meetings are being held virtually which has advantages but also many disadvantages. For example, the transmission of non-verbal signals becomes more difficult and changes our interaction behavior. People also report exhaustion caused by the huge number of video conferences, the so-called Zoom-Fatigue phenomenon. One issue is the constant self-view. Seeing yourself all the time is not natural and can have a positive effect on self-awareness, but, e.g., a negative one on enjoyment and perceived productivity. Therefore, alternatives to activated cameras that nevertheless transmit emotional reactions automatically could be useful. 
In our previous research, we used Facial Expression Recognition (FER) in video conferences to detect emotional states of participants. In these small-scale user studies, we analyzed the facial expressions of participants to detect and visualize emotions with a FER-tool and human observers. We discovered situations in which facial expressions alone were not sufficient to correctly identify emotions outside the lab in our small-scale study. Circumstances such as changing video quality or participant movement made automatic emotion recognition via faces challenging. In these scenarios, we could only analyze emotions when participants shared their videos during the meeting. 
In this paper, we would like to present a different approach, where we integrate and analyze a set of different physical signals in addition to facial expressions, and where people do not share their video in the conference, but an interface which changes depending on the emotional state of the participant. This makes it possible to share emotions without sharing the own video. For this purpose, we use our SitAdapt system. 
SitAdapt is an integrated software system for enabling situation-aware real-time adaptations for web and mobile applications. SitAdapt uses the different APIs of the devices such as eye-tracker, wristband, facial expression, and EEG signal recognition software, as well as metadata from the application to collect data about the user. The included rule editor allows the definition and modification of situation rules, e.g., for specifying the different user states and the resulting actions. The rule editor can use all input data types and attribute values as well as their temporal changes for formulating rule conditions. At the runtime of the application, the rules are triggered by the adaptation component for adapting the user interface, if the conditions of one or more rules apply. 

Benjamin Roszipal, Sebastian Egger-Lampl and Markus Karlseder: Enhancing Higher Education through XR Technology: An Innovative Approach to Knowledge Acquisition, Skill Development, and Examining

Knowledge acquisition and skill development are fundamental components of effective learning in higher education. Traditional methodologies involving pattern recognition, knowledge reproduction, and hands-on practice have long been utilized to build competencies in various fields. However, certain disciplines, such as healthcare and large industries, often face limitations in hands-on training due to safety concerns or resource-intensive simulations. 

This contribution explores the potential of experiential learning in digital simulations supported by Extended Reality (XR) technology as a novel and efficient approach to address these challenges. XR presents an advanced platform for creating immersive and engaging experiential learning experiences. Leveraging XR, we have developed a didactic concept that enables learners to acquire knowledge, develop skills, and achieve successful test results in a meaningful and effective manner. 

The implementation of XR supported experiential learning (XREL) offers several advantages over traditional methods. It provides learners with a safe and risk-free environment to practice complex tasks, promoting higher retention of knowledge and skill transfer. XREL can be tailored to diverse educational needs, catering to both beginners seeking foundational knowledge and seasoned experts looking to enhance their proficiency. 
Our research investigates the effectiveness of XREL in learning outcomes across various disciplines. We present a comprehensive analysis of our XREL-based didactic approach, demonstrating its applicability in upskilling employees, students, and experts alike. The study showcases how our XREL solutions facilitate experiential learning, empowering learners to acquire knowledge and skills through immersive experiences. 
Through this conference contribution, we aim to shed light on the potential of XREL technology in transforming higher education, bridging the gap between theoretical understanding and practical application. We provide empirical evidence of the positive impact of XR simulations on knowledge acquisition, skill development, and proficiency assessment. Ultimately, this work paves the way for a more efficient and accessible approach to learning and training in various fields, fostering continuous growth and advancement in education and professional development. 
XRCONSOLE is an R&D-driven SME focused on incorporating cutting-edge research into interactive educational XR technologies. Engaging in various national and international research projects, we develop XR prototypes, integrate behavioral probes and analytics to create educational and training KPIs. Our interdisciplinary team with scientific, development, and educational backgrounds ensures evidence-based practices, fostering meaningful collaboration with academia, industry, and educators. With a vision for immersive learning experiences, we strive to revolutionize education through XR technology, empowering learners of all backgrounds to acquire knowledge and skills in unprecedented ways. 

Dennis Rosenberg: Predictors of the Perceived Health-Related Usefulness of Mobile Devices in Later Life: Results of the Health Information National Trends Survey

Objectives and Research Question: The goal of this study was to assess the impact of the older adults' background characteristics on their perceptions of the health-related mobile device usefulness. The third-level digital divide approach was employed as the theoretical framework for the study. According to this approach, (older) people tend to benefit differently from using technology even when they have equal access to it and are equally competent or skillful in its use. The third-level digital divide approach maintains that the differences in these benefits or their perceptions are attributed to the categories (older) people belong. The study attempted to answer the following question: which socio-demographic and health-related characteristics of older adults are associated with their perception of mobile device usefulness in various health domains? Methods: The data were obtained from the Health Information National Trends Survey (Wave 5, Cycle 4) conducted in February-June 2020. The sample included 1373 U.S. older mobile device owners, i.e. people who reported owning either smartphone, tablet, or both. The lower limit for the respondent age, in accordance with the definitions of the United Nations and World Health Organization, was set at 60 years. The items served as dependent variables asked whether tablet or smartphone helped respondents in the following domains: tracking progress on a health-related goal (achieving health-related goal), making a decision about how to treat chronic illness or condition (medical decision making), and discussions with healthcare provider (patient-provider communication). Since each item was dichotomously introduced, the data were analyzed using logistic regression models. Results: Being 75 years old or older and self-definition as White significantly interacted with respect to perceived mobile device usefulness in achieving health-related goal and patient-provider communication. Age and White race variables were negatively and independently associated with perceived mobile device usefulness for medical decision making. Being married, female gender, better self-rated health, and having a hypertension diagnosis corresponded to a greater likelihood of perceiving own mobile device as useful in achieving health-related goal. Mental health disorder diagnosis positively related to perceived usefulness for medical decision making and patient-provider communication. Education and income levels also demonstrated some associations, mainly with respect to perceived usefulness in achieving health-related goal. Discussion: The findings provide support for the employed theoretical framework by showing that some categories of older adults are more likely than others to perceive their mobile devices useful in health domains. Health background seems to be less dominant than socio-demographic background in explanation of the mobile device health-related usefulness, providing additional support for the third-level digital divide approach. The findings may serve for preparation of programs aimed at encouraging older adults to use their mobile devices for health purposes. 

Robert Halwaß, Steffen Prowe, Evelyne Becker, Joachim Villwock, Martina Mauch, Linnea Pehl, Clara Simon and Lena Ziesmann: Immersive STEM-Laboratories to improve teaching and learning 

The project “Interactive Teaching in Virtual STEM labs | MINT-VR-Labs” (MINT is the German acronym for STEM) takes up the didactic potential of virtualizing laboratories in order to anchor innovative interaction formats in virtual space and thus implement new blended learning and virtual teaching/learning formats in practical modules for the Berlin University of Applied Sciences (BHT). In these virtual labs, students can, for example, explore biotechnology labs, look behind the scenes of a theater stage, learn basic programming skills in a pizza factory, or experience complex mathematical functions. Virtual laboratory exercises are intended to supplement classroom teaching, reduce the heterogeneity of students' prior knowledge, and increase student success. For this reason, the development and testing of these virtual laboratories will be didactically accompanied and the effectiveness of the use of these digital media in university teaching will be evaluated.

(Chair: Georg Vogt)