This September, Krzysztof Pietroszek visited Côte d'Azur University to begin facilitating the "Virtualization of the Subconscious" project, an innovative exploration of human consciousness through XR (Extended Reality) technologies. The project aims to digitally represent the subconscious mind by merging art with cognitive science, creating immersive experiences that allow participants to engage with the hidden layers of thought and emotion. Through this interdisciplinary approach, the project seeks to push the boundaries of how technology can visualize the complex workings of the mind.

The research objectives are centered around understanding and visualizing the subconscious in ways not previously possible. One key objective is to explore how immersive XR environments can be used to model and simulate subconscious thought processes. Another goal is to investigate how these digital representations might offer new tools for psychological research and therapeutic practices, potentially enabling breakthroughs in mental health treatments. The project also aims to examine the implications of engaging with the subconscious in a digital space, looking at how these interactions could reshape human-computer relations and enhance our understanding of cognitive processes.

Krzysztof Volumetric Specialist Becky Lake showcased our immersive experience, “Ethereal Motion; Dance in the Virtual Realm,” at the 2024 UFVA Conference as a New Media piece. “Ethereal Motion” features holographic performances of dancer and choreographer Rose Xinran Qi in the user’s choice of virtual environment. Conference participants marveled at the fun and expression provided by the different environments and performances. A few users mentioned they would love to stay and explore the virtual space beyond the time restraints of the conference. Becky also talked with other New Media creators about collaboration to further expand the reach of “Ethereal Motion.”

You can view Ethereal Motion and other UFVA New Media projects here.

Krzysztof Pietroszek witnessed the amazing Immersive Competition at this year’s Cannes Festival. Among the standout pieces were “En Amour” and “Human Violins,” both of which captivated audiences with their striking narratives and emotional depth. En amour (2024) is an immersive and interactive experience, at the crossroads of the performing arts, performance art and visual arts. This installation offers the chance to experience a symbolic metamorphosis around the themes of love, separation and transformation. Its narrative is rooted in an intimate story, made universal, to share a journey of sensations, emotions, shapes, colors, sounds and words that will awaken, and strengthen our love. HUMAN VIOLINS – PRELUDE is a visually striking interactive world evoking the power of transformative music in the darkest of times. Inspired by a heartbreaking real fact: during the Holocaust, many Jews were allowed to choose only one object before being brought to the camps and some chose their violin. We follow the fictionalized story of Alma (15), a young violin lover inhabiting a world of terror. Through music, she can liberate herself. You are offered the chance to take her legacy further.

Professor Pietroszek also praised the opening night’s French comedy, “The Second Act,” which revolves around the making of the first AI-directed movie. Overall, the high quality and artistic excellence of the entries in this year’s Immersive Competition reflect a significant advancement in the field, promising exciting future developments.

The 11th MINDCARE on Pervasive Technologies for Mental Health welcomed Krzysztof Pietroszek as a keynote speaker where he presented “Artificial Reality Continuum: Advancements and Dangers.” Pervasive Technologies are sensing technologies that collect various physical, social, and emotional data through sensors. The conference serves as a platform for professionals to showcase, learn, and explore topics like wearable computing, smart environments, and machine learning. Discussions center around crucial areas such as stress analysis, affective computing, and the integration of serious games into mental health. MINDCARE also provides a resource for young researchers to publish their work in the Sensors Journal, fostering collaboration and driving innovation in mental health research.

This 11 International MINDCATE was held in Milan on April 4 and 5, 2024.

Faculti invited Krzysztof Pietroszek to share an
overview of volumetric capture technology and its application to narrative filmmaking. With 3D recorded actors (volumetric capture), users are immersed in the story – a sharp contrast to the traditional mode of filmmaking. Additionally, in interactive and immersive filmmaking avenues, the participants rather than the Director, control the focus and direction of the story. You can watch the whole interview here. 

“The Faculti Streaming Platform gives users instant access to thousands of relevant and up-to-date academic and professional research insights annually.”

At CHIRA 2023, Krzysztof Pietroszek took on the role of Chair for a session focused on User Experience Evaluation. Throughout this session, Krzysztof guided discussions and presentations that delved into the forefront of Computer-Human Interaction (CHI) research. Among the presentations curated by Krzysztof was “Empowering Production Workers to Program Robots: A No-Code, Skill-Based Approach” by Charly Blanc, which focused on interactive devices. His leadership ensured that the session provided valuable insights into enhancing user experiences and evaluating emerging technologies in CHI.

Krzysztof Pietroszek also presented on a collaborative research project, “CPR Assistance in Mixed-Reality.” By using advanced technology like RGBD cameras and augmented reality overlays, the system offers real-time guidance to first responders during emergencies. Through thorough research and comparison with traditional video-based communication methods, Krzysztof and his team demonstrated the effectiveness of mixed reality in improving visual communication without overburdening the first responder.

Dr. Pietroszek presented on November 16, 2023

At EduLearn 2023, Professor Krzysztof Pietroszek showcased his NSF-funded research on Volumetric Communication in Medical Training. Over two years
of dedicated work, he and his team successfully developed, evaluated, and published a cutting-edge system that enables immersive and interactive training for healthcare professionals. Furthermore, Pietroszek’s expertise was recognized as he served as the chair for the Virtual and Augmented Reality session, further highlighting the growing importance of these technologies in the field of education.

Dr. Pietroszek and Manuel Rebol recently presented research at the 2023 Intelligent Tutoring Systems Conference (ITS2023). Their paper, “CPR Emergency Assistance Through Mixed Reality Communication,” offers a unique solution to improve remote assistance in the medical field. The system utilizes RGBD cameras to capture a three-dimensional view of the local scene, including the patient, the first responder, and the surroundings. The volumetric capture is then augmented onto the remote expert’s view, allowing them to provide spatial guidance using visual and verbal instructions.

By combining visual and verbal instructions in real-time, the system enables remote experts to guide first responders during life-saving procedures such as CPR. The research study confirmed the system’s effectiveness, highlighting the advantages of mixed reality over videoconferencing-based assistance. This innovative technology can improve CPR outcomes and significantly redefine emergency medical communication.

The success of this research study opens doors to further advancements in mixed reality communication systems, allowing experts to provide real-time guidance and support remotely, even in high-stress scenarios. As the technology continues to evolve, it could be integrated into various healthcare settings, emergency response training programs, and even everyday situations where immediate CPR assistance is required.

To honor the indomitable spirit of Polish mountaineering Wanda Rutkiewicz, the Institute for IDEAS recreated pivotal moments from Rutkiewicz's historical ascent of Mount Everest as holographic recordings. Three virtual and augmented reality scenes bring Wanda's awe-inspiring journey to life, transporting viewers into the world of one of history's most iconic female climbers.

One of the scenes depicts Wanda Rutkiewicz fearlessly traversing an ice-covered pass on her Everest expedition. The hologram showcases Wanda's careful steps across treacherous terrain, immersing viewers in her challenges and the breathtaking beauty surrounding her. Another scene catapults viewers into the heart-stopping moment when Wanda fearlessly scaled the icy precipice. 

A crucial aspect of recreating Wanda Rutkiewicz's climb lies in capturing the human experience amidst extreme conditions. The Institute for IDEAS encapsulates this by creating a scene depicting Wanda enjoying a meal in her tent before settling down for sleep. This intimate portrayal highlights the significance of sustenance and rest in the demanding world of high-altitude mountaineering. Viewers are transported into the tent, sharing a moment of respite with Wanda as they witness the importance of self-care and preparation for the next leg of her journey.

Through these holographic scenes, Wanda's incredible feats become accessible to a global audience, fostering a deeper understanding of her courage, perseverance, and trailblazing spirit. These holograms exemplify the Institute's commitment to pushing the boundaries of immersive technologies, enabling individuals to connect with and be inspired by the remarkable achievements of those who came before us.

The Institute for IDEAS collaborated with the exceptionally talented dancer and choreographer Rose Xinran Qi. This groundbreaking partnership involved capturing and rendering holographic dance performances with our state-of-the-art volumetric studio. The result was an awe-inspiring visual experience, where viewers could witness her dance as if she were physically present before them. As a result, Rose Xinran Qi's visit to the studio marked a significant milestone in exploring the intersection between dance and holographic technology.

The collaboration between Rose Xinran Qi and presents us IDEAS has provided invaluable insights into holographic dance performances' possibilities and potential applications. This research endeavor aims to revolutionize the way dance is experienced, not only by audiences but also by the dancers themselves.

The Institute for IDEAS intends to utilize the data and knowledge gained from this collaboration to explore various applications of holographic dance performances in augmented and virtual reality. This technology opens up new avenues for dance education, where aspiring dancers can learn from the holographic projections of esteemed performers. Moreover, choreographers can experiment and create stunning visual spectacles by blending physical and holographic dancers. 

Associate Director, Bei Xiao, received a $420,000 research grant from the National Institute of Health for he project "Learning diagnostic latent representations for human material perception: common mechanisms and individual variability". The project starting date is April 1st, 2023.

Learn More

The Institute hosted Maryland-based musical duo Dunia & Aram, composed of ska-punk performer Dunia Best and AU professor Aram Sinnreich. Dunia & Aram performed their arrangement of Billie Holiday's heart wrenching classic "Strange Fruit", a song from their new album, Bedfellows. The recordings will be used to create a 360° music video featuring multiple holograms of Dunia and Aram performing all the instruments and vocals for the song.

Learn More

Last month, Becky Lake and Meagan Couture attended the University Film & Video Association Conference to represent their project Space Ocean Library, a collaboration with American University students and the Institute for IDEAS. Users tested the experience and provided feedback for future development.

Learn More

The Institute for IDEAS' cutting-edge volumetric capture studio recently played a central role in the development of a groundbreaking documentary research project at the American University of Sharjah. In collaboration with Professor Suheil Dahdal, the Institute produced several holograms for use in the documentary experience/research initiative Emirati Stories AR Documentary.

Learn More

On the 20th of June, Institute for IDEAS founder Dr. Krzysztof Pietroszek hosted a meeting at the prestigious TAI School of the Arts in Madrid, Spain. This discussion, titled "Narrative Uses Of Volumetric Capture: Storytelling And The Artificial Reality Continuum", explored the potential uses of volumetric capture through cutting edge technology such as the Institute's Tetavi capture studio.

Volumetric capture brings real world performers and objects seamlessly into the virtual space, using photogrammetry. In his talk, Dr. Pietroszek addressed the ramifications of this technology for filmmaking and other arts, as well as the many avenues for expansion within this medium, like haptic sensory experiences. Volumetric capture is an innovation in digital media, and the Institute for IDEAS is honored to collaborate with TAI in pioneering this new field of the arts.

Based out of a high-tech studio space, the Institute is developing immersive volumetric content for VR, AR, and other applications. The core of this initiative is the Institute's volumetric capture system, purchased with a $1 million grant from the National Science Foundation. TDesigned by international tech company Tetavi, the volumetric capture studio smoothly transforms recorded performances into high-quality holograms. Tetavi's groundbreaking algorithmic process delivers a detailed, animated model of the subject for deployment across a number of platforms.

Learn More

Once a domain of science fiction, holographic filmmaking is becoming the next technological revolution. At the 2021 Cannes Film Festival, Dr. Pietroszek voices the capabilities and complexities of the evolving medium. He shares how storytellers are now using volumetric capture to recreate objects in virtual spaces. Once recorded, these holograms can then be placed in virtual or real environments, creating an experience Dr. Pietroszek compares it to Star Trek’s Holodeck.

Actors, props, sets, and even entire structures can now be captured as photorealistic 3D models, shown as holograms; This process is known as photogrammetry. Photogrammetry allows the extraction of 3D data from photos and drawings. Once captured, audience members can view these elements through head-mounted glasses, known as a cave system. This cave system allows for an immersive experience where one can enter and interact with the virtual world. Some bring audience members into the set, others bring props and set elements into the real world. Like computer coding, the language and grammar of volumetric filmmaking can be very complex. 

While the technology is ready for new stories and hniques, our understanding of the language and grammar of this medium is very limited. Through the Cannon keynote, Dr. Pietroszek discusses how filmmakers can learn and use volumetric filmmaking. Using low-cost volumetric capture technologies, Dr.Pietroszek invites filmmakers to experiment with volumetric capture. “Holographic filmmaking is a new and revolutionary form of storytelling that will not replace traditional cinema, but rather, it will become yet another” Dr. Pietroszek.

Dr. Pietroszek presented July 9, 2021

The holographic telehealth project of the Institute for IDEAS (AU) and the School of Medicine (GW) will transform the way medical personnel communicates and collaborates across the distance by allowing for real-time exchange of three-dimensional information that is missing in current videoconferencing telehealth. The project will lead to more equitable access to healthcare and will improved success for medical procedures that require the assistance of a remote expert.

There are a wide variety of artificial intelligence (AI) algorithms designed to make decisions for a number of different real-world problems. The purpose of this project is to systematically understand the difficulty that AI challenge problems pose to standard decision-making algorithms, as well as how robust such conclusions are to variations in problem design, problem size, computational resources, and algorithm configuration.