iiinova — where your digital self keeps evolving

Jeffrey P. Bigham @jeffreybigham.com · 16/09/2026

i'm still useful! or, maybe it's just being sycophantic?? :(

240

Jeffrey P. Bigham @jeffreybigham.com · 10/09/2026

write your paper with AI, then use this service to make it sound like a human again, hahaha.

010

Jeffrey P. Bigham @jeffreybigham.com · 22/08/2026

whoa, slow down a bit delta …

elevate your next flight, 26 drink vouchers

290

Jeffrey P. Bigham @jeffreybigham.com · 25/07/2026

good to see this focus on asking clarifying from claude, lots of interesting questions about how to manage human agency while benefiting from automation -- choices is a direction we've been pursuing, e.g., in Morae and CowPilot -- www.cs.cmu.edu/~jbigham/pub... aclanthology.org/2025.naacl-d...

Judgment
It more regularly asks clarifying questions before guessing, pushes back on flawed instructions, and considers the implications of its work before jumping ahead to implementation. The result is cleaner, higher-quality code.

130

Jeffrey P. Bigham @jeffreybigham.com · 16/07/2026

as an author, I should obviously look over this paragraph, but it has basically just filled in filler words around the topics and details I gave it, I don't see major hallucinations, seems pretty reasonable, but YET it's classified as 100% AI generated

100

Jeffrey P. Bigham @jeffreybigham.com · 16/07/2026

I think about appropriate AI assistance in writing based on what human workflows are supported. from that way of thinking, it's surprising to me that NeurIPS used Pangram to reject papers that were "AI generated" b/c of the legit use cases it rejects -- consider bullet points to paragraph, e.g.,

200

Jeffrey P. Bigham @jeffreybigham.com · 18/05/2026

this looks great for pittsburgh, and it is better than most places. i will say that so much of the housing stock in Pittsburgh is very old and poor quality, so there are lots of houses on the market for literally $20k. you do not want anyone to live in these houses. www.nytimes.com/interactive/...

121

Jeffrey P. Bigham @jeffreybigham.com · 14/05/2026

my teenager wants to get a job helping run camps at her school, these are all the forms she has to fill out lol

140

Jeffrey P. Bigham @jeffreybigham.com · 23/04/2026

is Pittsburgh’s draft nonsense a bust? Hotel prices are going down, not up, for this weekend, and seem like a pretty normal to below average cost

030

Jeffrey P. Bigham @jeffreybigham.com · 22/04/2026

kind of weird that supposedly 700,000 people are coming to Pittsburgh for the NFL Draft, and yet you can get a hotel for $180/night for Friday->Sunday

010

Jeffrey P. Bigham @jeffreybigham.com · 17/04/2026

nice of the Barcelona Airport to make me a sign

jeff in front of sign that says, "Best Ham in the Airport"

180

Jeffrey P. Bigham @jeffreybigham.com · 06/04/2026

too many people got the brain rot! and i don't blame AI, i blame YOU

000

Jeffrey P. Bigham @jeffreybigham.com · 23/03/2026

wow, they are apparently wearing masks even at airports --

020

Jeffrey P. Bigham @jeffreybigham.com · 21/03/2026

they say it's impossible to cross party lines these days. but chuck norris does it. fitting. RIP

010

Jeffrey P. Bigham @jeffreybigham.com · 08/03/2026

redoing my CV and such, so far a fun experience with Claude Code

Update Todos

Discover local PDFs from cs.cmu.edu/~jbigham

Look up DOIs for publications missing them

Add publisher links (ACM DL, etc.)

Update BibTeX with enriched metadata

Update templates to show new links

Rebuild and verify all formats

271

Jeffrey P. Bigham @jeffreybigham.com · 07/03/2026

www.post-gazette.com/business/tec...

As of Friday, only seven companies are certified to operate “Highly Autonomous Vehicles” with a driver on board in Pennsylvania: Aurora, Motional, Stack, CMU, Perrone Robotics, Mapless AI and Waymo.

010

Jeffrey P. Bigham @jeffreybigham.com · 19/01/2026

this photo, taken 18 years ago today in the Canary Islands for IUI 2008, is still my profile picture in a lot of places 😂

160

Jeffrey P. Bigham @jeffreybigham.com · 15/01/2026

i reference this sticker vending machine from the local ice cream shop surprisingly often -- it provides rotating 22-set curations of "butts on things" providing this photograph in case it might be useful to you in your life

butts on things sticker vending machin, shows burrito with a butt, beaker with a butt, watermelon butt, etc.

110

Jeffrey P. Bigham @jeffreybigham.com · 13/01/2026

when your wife is friends with a romance novelist and you get an amazon alert about delivery of their latest book … lol

Out for delivery: Arriving today 11am-1pm: "The Lust Crusade" and

000

Jeffrey P. Bigham @jeffreybigham.com · 24/12/2025

honestly, they seem pretty good. they often seem more "accessible" (approachable) than my own abstracts, usually seem accurate. it's related to how tweet threads about papers are always better/more compelling than abstracts, b/c of weird publication norms/expectations.

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app’s full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.

# Summary

This research addresses the long-standing problem of inaccessible mobile applications by developing an automated system to generate accessibility metadata directly from app screenshots. Many iOS and Android apps lack sufficient semantic information needed for accessibility features like screen readers to function properly, leaving people with visual impairments unable to use them effectively.

The researchers created a comprehensive dataset of 77,637 annotated screens from 4,068 iPhone apps to train their system. They collected screenshots and UI element information, then had workers manually annotate visual UI elements with detailed classifications and attributes. This dataset revealed significant accessibility gaps: 59 percent of screens had UI elements not exposed to accessibility services, and 94 percent of apps contained at least one such screen.

The core of their approach is an on-device object detection model trained to identify UI elements directly from pixels. They selected a Single Shot MultiBox Detector architecture using MobileNetV1 to balance accuracy with performance constraints, achieving 71.3 percent mean average precision on their test set. The model uses only 20MB of memory and requires just 10 milliseconds for inference, making it practical for real-time use on iPhones.

Beyond basic detection, the researchers implemented several augmentations to provide complete accessibility metadata. They added heuristics for grouping related elements, determining navigation order, and detecting UI states like selected or unselected toggles. They integrated iOS built-in features for optical character recognition, icon recognition, and image descriptions to provide content information. They also developed a separate model to predict whether icons are clickable with high precision.

To evaluate the approach, they conducted user studies with nine experienced screen reader users who tested inaccessible apps using both standard VoiceOver and their Screen Recognition system. Results showed significant improvement: usability ratings averaged 2.08 for regular VoiceOver compared to 3.73 with Screen Recognition, a statistically significant difference. Participants reported becoming aware of previously hidden UI elements and successfully using apps they had previously found completely unusable.

The system called Screen Recognition generates this metadata at runtime and makes it available to iOS VoiceOver, enabling accessibility even for apps without developer-provided metadata. While limitations remain, such as difficulty understanding complex gestures without visual affordances and occasional performance lag on complex screens, the approach demonstrates a viable pathway to dramatically improve accessibility for existing mobile applications without requiring developer cooperation. The researchers plan to extend this work to other platforms and contexts beyond mobile, and to integrate it into developer tools to help create more accessible apps from the start.

AI-Generated Summary (Experimental)
This summary was generated using automated tools and was not authored or reviewed by the article's author(s). It is provided to support discovery, help readers assess relevance, and assist readers from adjacent research areas in understanding the work. It is intended to complement the author-supplied abstract, which remains the primary summary of the paper. The full article remains the authoritative version of record. Click here to learn more.

The summary is generated through a controlled process that uses a third-party service operating under contractual restrictions. Article content is used only for the purpose of generating the summary and is not used for model training, retained for reuse, or incorporated into other systems.

Click the blue "Feedback" button on the right side of this page to comment on the accuracy, clarity, and usefulness of this summary. Doing so will help inform refinements and future regenerated versions.

110

Jeffrey P. Bigham @jeffreybigham.com · 10/12/2025

voicemail transcription, any idea what it originally said?

220

Jeffrey P. Bigham @jeffreybigham.com · 08/11/2025

it happened, the vizwiz challenge surpassed the original vizwiz paper by citations --

Vizwiz grand challenge: Answering visual questions from blind people
D Gurari, Q Li, AJ Stangl, A Guo, C Lin, K Grauman, J Luo, JP Bigham
Proceedings of the IEEE conference on computer vision and pattern …
1193 2018
Vizwiz: nearly real-time answers to visual questions
JP Bigham, C Jayant, H Ji, G Little, A Miller, RC Miller, R Miller, ...
Proceedings of the 23nd annual ACM symposium on User interface software and …
1153 2010

060

Jeffrey P. Bigham @jeffreybigham.com · 30/10/2025

in 2014, I bought 3 nest thermostats for my house, which cost like $600… 10 years later, google apparently can’t afford to keep the api running, but had offered me an option to pay $250 per thermostat that will do exactly the same thing as my old one #internetofshit

2110

Jeffrey P. Bigham @jeffreybigham.com · 29/10/2025

i don't think my plane is gonna fit in this tiny little airport? what is that, an airport for ants?

visuazliation of my flight approaching denver, which for some reason has the plane an enormous size (bigger than all of denver)

050

Jeffrey P. Bigham @jeffreybigham.com · 07/10/2025

some beautiful fall weather for pittsburgh, which is showing off into the second week of october --

temperatures for the next five days range from 42 low to 67 high, sunny every day!

010

Jeffrey P. Bigham @jeffreybigham.com · 29/09/2025

#3 paper at #uist2025, by @mlam.bsky.social from her Apple internship, on Policy Maps, which is a visualization tool to help map out the unbounded space of LLM behaviors Wednesday, 11:24am in Miami www.cs.cmu.edu/~jbigham/pub...

Policy maps chart LLM policy coverage over an unbounded space of model behaviors. Here, an AI practitioner is
designing a policy for how an LLM should summarize violent text. Policy map abstractions (right) allow the policy designer to
interactively author and test policies that govern a model’s behavior using if-then rules over concepts. The designer can create
any desired concept by providing a simple text definition to capture cases of model behavior. Our Policy Projector tool (center)
renders cases, concepts, and policies as visual map layers to aid iterative policy design.

060

Jeffrey P. Bigham @jeffreybigham.com · 29/09/2025

#2 paper, paper by my student @yihaopeng.bsky.social, on Morae which is a web agent that is accessible to screen reader users, and also pauses when there are choices to be made see a demo tonight/Monday, and the talk Wed 2:42pm, Sydney www.cs.cmu.edu/~jbigham/pub...

Morae is an accessible user interface agent that proactively pauses automation at key decision points for blind and
low-vision users to make choices. For example, when asked to “buy the cheapest sweetened sparkling water”, existing agents
would automatically choose a product even if multiple items have identical prices. Instead, Morae pauses automation and
presents relevant product differences (e.g., flavors, ratings) and allows users to choose based on their preferences. By proactively
detecting and pausing when user input is necessary, Morae allow users to actively express preferences during UI automation.

140

Jeffrey P. Bigham @jeffreybigham.com · 29/09/2025

I have three papers at #uist2025 -- #1. StepWrite, how can we compose long-form text with speech -- works better than chatGPT and dictation Tuesday, 2:54pm, Capri room www.cs.cmu.edu/~jbigham/pub...

example of hands busy scenario, writing an email, by answering targeted questions generated by the LLM-powered system

140

Jeffrey P. Bigham @jeffreybigham.com · 29/09/2025

i'm at #uist2025 this week in beautiful Busan, Korea -- yesterday, i got to serve on the Doctoral Consortium panel, amazing to see the great work of 10 rising star students in technical HCI … and look at this amazing lunch! :)

1200

Jeffrey P. Bigham @jeffreybigham.com · 08/09/2025

recognizing customer demand, or at least fake internet outrage, my research group has reverted our logo to the original. regardless of our logo, we have always remained committed to delicious homestyle HCI systems and accessibility research like my grandma used to make.

1110

Jeffrey P. Bigham @jeffreybigham.com · 10/08/2025

highway is closed today, as a warmup to next summer when it'll be closed for a whole month… google maps is directing folks through the tiny winding neighorhood/park roads, "official detour" is nowhere to be found… already a mess on a sunday, can't even imagine how bad this is gonna be!

021

Jeffrey P. Bigham @jeffreybigham.com · 06/08/2025

tired: genAI is fine wired: we need to build whole massive efforts to help ensure that genAI doesn’t tell people to eat rocks inspired: let’s put AI in nuclear weapons, for some reason that we’ll figure out later

170

Jeffrey P. Bigham @jeffreybigham.com · 06/08/2025

100% -- as a quite seasoned runner, including in cities, there are many places where i will avoid the intersections, as weird as that might sound. cars can be MORE distracted at intersections than they are on the straight parts of the road.

But pedestrians navigating a landscape built for cars rightly perceive intersections as unsafe, too. On West Hudson, pedestrians who have pressed the button for a walk signal still must contend with fast-moving cars turning as they cross.

220

Jeffrey P. Bigham @jeffreybigham.com · 06/08/2025

i'm all in favor of calling out where tariffs are costing us additional money, but of course businesses are using them as another way to just sneak random undisclosed fees onto the final receipt … i guess it's for the shirt (or the glass i bought), not the actual run haha

120

Jeffrey P. Bigham @jeffreybigham.com · 01/08/2025

a few things differentiate me from JD Vance, in addition to me having more appalachian bonafides than he does, … I grew up never questioning what was possible… so, when i achieved things, it seemed normal I always thought this was the rural strength, he accepts it as a foregone weakness

Archived clip of Vance: I realized that in this new world, I was the cultural alien. I began to think seriously about questions that had nagged at me since I was a teenager: Why has no one else from my high school made it to the Ivy League? Why are people like me so poorly represented in America’s elite institutions? Why is domestic strife so common in families like mine? Why did I think that places like Yale and Harvard were so unreachable? Why did successful people feel so different?

140

Jeffrey P. Bigham @jeffreybigham.com · 12/07/2025

they should put these absurd requirements directly on the box, I don’t even have ONE capable adult…

1111

Jeffrey P. Bigham @jeffreybigham.com · 09/07/2025

new nomination for "internet of shit" -- google is withdrawing support for my Nest thermostat (that works perfectly fine), but has kindly offered me the opportunity to pay $150 for a new one that will (ideally) do the same thing =

080

Jeffrey P. Bigham @jeffreybigham.com · 06/07/2025

not entirely sure why i keep doing this to myself, but figured I could probably act like an AC… lol … also @barik.net promised me I wouldn't have to sit on a 6 hour PC meeting phone call where I have no opportunity to do anything and nothing is actually decided. didn't you Titus?

Jeffrey Bigham -

Thank you for agreeing to act as a committee member for CHI 2026 Papers.

260

Jeffrey P. Bigham @jeffreybigham.com · 21/06/2025

Come on Carol! (adventures in old signs)

carol never wore her safety goggles, now she doesn’t need them

020

Jeffrey P. Bigham @jeffreybigham.com · 05/06/2025

ok, actually, this is much scarier …

251

Jeffrey P. Bigham @jeffreybigham.com · 05/06/2025

ChatGPT is getting pretty good! Although, now we’re back to copy pasting probably, but I love how it guides them to describe what about the paper they like

130

Jeffrey P. Bigham @jeffreybigham.com · 19/05/2025

(somewhat random) nytimes op-ed linked to @hcii.cmu.edu's "Human-AI Interaction" course that i started long ago, as an example of the kind of things we should be teaching and, sure, it's not a "bachelor program", but an individual course. www.nytimes.com/2025/05/19/o...

First, we need to ensure workers are learning the skills employers are starting to demand. New approaches are emerging: American University’s Kogod School of Business is embedding A.I. across its curriculum and training faculty members to use A.I. tools, while Carnegie Mellon is offering an A.I. bachelor’s program in which students take classes on harnessing the power of A.I. to be “beneficial and useful for people.” Community colleges in Miami-Dade, Houston and Maricopa Community Colleges are joining in, starting a national A.I. consortium to align curriculums with work force needs and offer applied A.I. degrees backed by companies like Intel and Microsoft.

080

Jeffrey P. Bigham @jeffreybigham.com · 11/05/2025

90s evening fundraiser with my lady

280