iiinova — where your digital self keeps evolving

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 05/08/2025

Over the last two years we have seen an acceleration on AI, but if AI is going to enable humans in their day to day tasks, it most probably will be via XR. The issue then is if a selection has real-world consequences, we will need great precision to interact.

191

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 28/04/2025

This precisely was something we tried to address and quantify in the VALID avatars. We found that ingroup participants were much better at determining the ethnicity of the avatars. And outgroups just had low accuracy labeling MENA, AIAN, NHPI and Hispanics. github.com/xrtlab/Valid...

060

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 25/03/2025

But really as it happens with all datasets, the key is on “How to make data useful?” That was our goal with PARSE-Ego4D parse-ego4d.github.io we augment the existing narrations and videos by providing 18k suggested actions using the latest of VLMs. This can be useful to train a future AI agents.

111

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 23/03/2025

But also: Not all avatars are created equal ⚠️. We did a deep study on preferences (presented at ISMAR 2023 arxiv.org/pdf/2304.01405) and realism matters quite a lot for work scenarios. most people don’t find acceptable a cartoon avatar…

010

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 13/03/2025

More over... we put all of this in an interactive dataset for people to explore each of the techniques and geek around😎 xrtexttrove.github.io some are wild!

140

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 13/03/2025

3) there is a tendency to think custom hardware will solve this gap but so far it hasn’t. And keyboard format still delivers the highest wpm.

110

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 13/03/2025

2) external surfaces and fingertips/transparency of hands seem to be preferable as ways to improve performance compared to midair gestures.

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 13/03/2025

1) vr text input has a ceiling effect for as long as we only use 1 or 2 fingers we will never get better than hunt and peck methods. * more or less we find that every additional finger used increases 5 wpm (words per minute). Users of multi finger solutions (touch typers) can reach 40-50 wpm.

120

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 13/03/2025

Ever wondered why is so hard to write in VR/XR? We too 😅. Check this new work to appear in ACM CHI: xrtexttrove.github.io We (@arpitbhatia.bsky.social et al) analyzed 176 papers and text entry techniques inside XR and found that:

1182

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 10/03/2025

Also look at this: “mundus sine cesaribus”, the CEO of bluesky

000

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 06/03/2025

We will also present “Beyond the Phone: Exploring Context-Aware Interaction Between Mobile and Mixed Reality Devices" by Fengyuan Zhu et al. showcasing the many interesting ways in which we can bring our phones inside VR. Paper:...

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 06/03/2025

At @IEEEVR we present “EmBARDiment: an Embodied AI Agent for Productivity in XR" by @riccardobovoHCI et al. showing how powerful AI agents can be when they are aware of the content around us, driven by contextual inputs like gaze. Web: embardiment.github.io Code: soon

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 06/03/2025

020

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 06/03/2025

Next week at #IEEEVR we present “EmBARDiment: an Embodied AI Agent for Productivity in XR" by @riccardobovo.bsky.social et al. showing how powerful AI agents can be when they are aware of the content around us, driven by contextual inputs like gaze. Web: embardiment.github.io Code: coming soon

1131

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 01/01/2025

I guess sometimes when we migrate we bring a baggage too.. when @davledo.bsky.social and @fbrudy.bsky.social invited me to talk at @autodeskresearch.bsky.social he prepared a super cute visual summary that deserves porting ;) challenges stand stronger now that we kind of tackled the OS in the room 🐘

040

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 30/12/2024

And prototyping full scenarios of future interactions. Like the work with Lystbaek et al. “Hands-on, Hands-off: Gaze-Assisted Bimanual 3D Interaction” UIST 2024 pure.au.dk/ws/portalfil...

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 30/12/2024

But the work had many legs also into top academic labs, let’s dive into that: For hand + eye. We worked with Profs Hans Gellersen and Ken Pfeuffer for 2 years. Figuring out design considerations: research.google/pubs/design-...

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 30/12/2024

Here are some of the topics, (but we are open to more :)): - Adaptive and Context-Aware AR - Always-on AI Assistant by Integrating LLMs and AR - AI-Assisted Task Guidance in AR - AI-in-the-Loop On-Demand AR Content Creation - AI for Accessible AR Design - Real-World-Oriented AI Agents

030

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 30/12/2024

Ryo, Misha, David and myself are organizing a workshop at #CHI2025 on *Everyday AR through AI-in-the-Loop*. If you are interested in topics such as adaptive AR, AR+AI Agents, or task guidance, consider submitting. We are looking for 1-2 pagers! Deadline is Feb 15th. xr-and-ai.github.io/

192

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 30/11/2024

Missinformation just an oversight of integrity. For scientific studies it looks like the horoscope: “if you eat eggs you live longer”, but “you should cut eggs if you want longevity”. It’s nefarious for science. But IMO journalism lack of integrity is worse on the nefarious axis.

000

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 23/11/2024

Incredible line up of talks at the #FutureOfText Symposium. Thanks so much to the organizers @liquidizer @dgrigar. In the pics: Ge Li from @DSE_Unibo, @KenPfeuffer from @AarhusUni_int and Ryan House from @UWMadison. Great support by @SloanFoundation and @wsu.

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 22/10/2024

Great @hrvojebenko!! Impressed by the variety of wrist prototypes that you have built over the years. Your work has come a long way since our time at MSR. I ❤️the end of the keynote, quoting @wasbuxton, also a former MSR colleague.

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 22/10/2024

This week at @ismarconf we will present two foundational workpieces on Avatar Collaboration. #AvatarPilot and #MRTransformer. As we add ++ people to a VR space the chances of their spaces overlapping tends to Zero. This work solves N people collaborating in dissimilar spaces. 🧵

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 21/10/2024

@IEEEmetaverse symposium is kicking up. Wonderful line up of talks. Looking forward to all the discussions.

000

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 11/10/2024

Further, we propose a simple kernel-based score for detecting continuity issues across image borders of 2D projections of spherical images (horizontal field of view).

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 11/10/2024

FID has crucial shortcomings for evaluating spherical images. We show that we can manipulate the vertical field of view, distorting the geometry of the images, with negligible increase in FID 👎 We propose using cubemap projections to increase compatibility with the CNN backbone

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 11/10/2024

This framing of Geometry Fidelity tackles also a well known problem in the AI community: where we have focused a lot on training & inference and much less on the evaluation (horse meme). But for new formats like 360 images, we need to put the zoom on evaluation.

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 25/09/2024

We explored applications, and did a user study based on phone @GoogleARCore, and HMD. Some of them are showcased in our paper: M Doga Dogan, E J. Gonzalez, K Ahuja, R Du, A Colaço, J Lee, M Gonzalez-Franco, D Kim (2024) "Augmented Object Intelligence with XR-Objects" @ACMUIST

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 25/09/2024

The architecture of XR-Objects is very similar to how we are seeing new AI arriving to users. Beyond chatbots, there is still a lot of system engineering making AI more interactive and linking pieces of Computer Vision, LLMs and Computer Graphics.

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 25/09/2024

XR-Objects envisions a world where Analog objects can be interacted like digital thanks to AI, by enabling Augmented Object Intelligence (AOI).

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 25/09/2024

Like many of you I was👀the launch of Orion #MetaConnect. I❤️seeing @finkd showcase this application. So in line with our XR-Objects vision. While the dev kit rolls out, you can try our opensource xr-objects.github.io. Here findings of our explorations:🧵

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 21/08/2024

So this is what we did: Gonzalez-Franco M, Steed A, Berger CC, Tajadura-Jiménez A (2024) The impact of first-person avatar customization on embodiment in immersive virtual reality. Front. Virtual Real. 5:1436752....

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 22/07/2024

Part of the problem for immersive generative content is that often we rely on small datasets that hinder training. To achieve better results we must improve the way we evaluate. Here we introduce OmniFID to improve evaluation for spherical images....

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 11/07/2024

Looking forward to this Hispanic/Latino discussion with the @arxiv community on their annual Accessibility Forum. accessibility2024.arxiv.org Con muchas ganas de participar en el foro accesibilidad e acessibilidade de @arxiv (La conversación será en Español y Portugues)

000

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 22/06/2024

After AWE, @marknb00 got to visit our lab in Seattle, and discuss about how different was it all when he did his PhD at @UW, in the early HIT lab days with @electrontom. (Here Mark checking out the future with @_VascoXu).

000

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 22/06/2024

Also went to AWE @ARealityEvent and met (in person) with my favorite bloggers @SkarredGhost @hmltn and non bloggers @klukoff @xrmasiso @eandreabravo

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 22/06/2024

Then hanged out with @zsoltkira at @CVPR, celebrated the wonderful reception our DiffSeg paper had sites.google.com/view/diffseg/home @lavishaaggarwal killed it on the presentation :) We are already devising what’s next.

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 22/06/2024

The last two weeks have been busy with illustrious visitors at the BIRD lab 🧵. A memorable lunch with @hciprof by the water canals of Fremont about best form factors to embed AI in devices (@danmcduff, @_ejgonz_, @realkaranahuja maybe this is it? 🤪)

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 04/06/2024

I will be @ARealityEvent this year🎉! I have been debating for a while whether to join or not a Consumer Conference, as I normally only go to Scientific venues. I will try to bridge the gap between the two communities with my talk.❤️@GoogleARVR...

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 19/05/2024

Last two weeks we have had some illustrious visits at BIRD. On their way to #CHI2024 @HansGellersen @KenPfeuffer & Mathias stopped for a small but great workshop day. Shortly after @haraldsson came for lunch: looking fwd to doing more w @xrcollaboratory, @csaudk & @HCILancaster

000

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 11/05/2024

@andcee and I focus on information workers. Who already have a lot of devices: VR just one more! A big part of the paper delves into shifting the question of what’s the “killer app” into “killer medium”, where we can do focus, transitions and multitasking. Proper OS support work.

000

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 09/05/2024

Sorry reviewer, we didn’t change the teaser, because look at the playful scenarios of XR with our existing devices if we know intent. Video: youtu.be/r7fX2LUx3xE?si=Mr-0Pf3_PAi… @_ejgonz_ @realkaranahuja @andcee @ishanchatterjee

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 06/05/2024

Seattle team 🦜 offsite at the most beautiful BIRD cage: the public library @SPLBuzz designed by @PritzkerPrize Rem Koolhaas.

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 02/05/2024

Cool things on this new XR gaming paper. 1) We compiled a very comprehensive questionnaire including presence, social presence, co-presence, enjoyment, embodiment etc with q directed to the VR players, observers and other players and 2) asked experts about their experiences.

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 02/05/2024

We played a couple of different games that allow asymmetric play mostly @DavigoGame @AcronGame VR+PC, VR+phone, highly recommended for parties by the way!. An interesting thing is that the different mediums afford asymmetry also on skills - check the full paper we go deep!

100

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 02/05/2024

Very excited about the two LBW that the BIRD team and collaborators will present at #CHI2024 The first paper is about asymmetric play in XR. What happens when one person is wearing a headset and the rest aren’t? (Inspired during our @dagstuhl in social XR late last year). 🧵

110

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 12/04/2024

New 🦜 BIRDs that have joined us in the last couple of months. @JamesBHou from @HCILancaster and @_VascoXu from @UChicago. Welcome! We are so excited to have you join us in Seattle.

000

Dr. Mar Gonzalez-Franco @twimar.bsky.social · 04/04/2024

Happy 500 ⭐️day to the #MicrosoftRocketbox avatars github.com/microsoft/Microsoft-Rock… This baby was born pre-pandemic, and had a family taking care of them. From the original creators, to the custodians at Havok. To the archeology I had to do to revive them. ...

100