Sign in

Jeffrey P. Bigham

@jeffreybigham.com
4.9K followers 913 following 1.4K posts

Professor of HCII and LTI at Carnegie Mellon School of Computer Science. jeffreybigham.com

PostsRepliesMedia
Jeffrey P. Bigham @jeffreybigham.com · 25/09/2026
let's go carnegie tech!
CMU #1 in computer science, #3 in graduate earnings
020
Jeffrey P. Bigham @jeffreybigham.com · 16/09/2026
i'm still useful! or, maybe it's just being sycophantic?? :(
240
Jeffrey P. Bigham @jeffreybigham.com · 10/09/2026
write your paper with AI, then use this service to make it sound like a human again, hahaha.
010
Jeffrey P. Bigham @jeffreybigham.com · 22/08/2026
whoa, slow down a bit delta …
elevate your next flight, 26 drink vouchers
290
Jeffrey P. Bigham @jeffreybigham.com · 25/07/2026
good to see this focus on asking clarifying from claude, lots of interesting questions about how to manage human agency while benefiting from automation -- choices is a direction we've been pursuing, e.g., in Morae and CowPilot -- www.cs.cmu.edu/~jbigham/pub... aclanthology.org/2025.naacl-d...
Judgment
It more regularly asks clarifying questions before guessing, pushes back on flawed instructions, and considers the implications of its work before jumping ahead to implementation. The result is cleaner, higher-quality code.
130
Jeffrey P. Bigham @jeffreybigham.com · 16/07/2026
as an author, I should obviously look over this paragraph, but it has basically just filled in filler words around the topics and details I gave it, I don't see major hallucinations, seems pretty reasonable, but YET it's classified as 100% AI generated
100
Jeffrey P. Bigham @jeffreybigham.com · 16/07/2026
I think about appropriate AI assistance in writing based on what human workflows are supported. from that way of thinking, it's surprising to me that NeurIPS used Pangram to reject papers that were "AI generated" b/c of the legit use cases it rejects -- consider bullet points to paragraph, e.g.,
200
Jeffrey P. Bigham @jeffreybigham.com · 18/05/2026
this looks great for pittsburgh, and it is better than most places. i will say that so much of the housing stock in Pittsburgh is very old and poor quality, so there are lots of houses on the market for literally $20k. you do not want anyone to live in these houses. www.nytimes.com/interactive/...
121
Jeffrey P. Bigham @jeffreybigham.com · 14/05/2026
my teenager wants to get a job helping run camps at her school, these are all the forms she has to fill out lol
140
Jeffrey P. Bigham @jeffreybigham.com · 23/04/2026
is Pittsburgh’s draft nonsense a bust? Hotel prices are going down, not up, for this weekend, and seem like a pretty normal to below average cost
030
Jeffrey P. Bigham @jeffreybigham.com · 22/04/2026
kind of weird that supposedly 700,000 people are coming to Pittsburgh for the NFL Draft, and yet you can get a hotel for $180/night for Friday->Sunday
010
Jeffrey P. Bigham @jeffreybigham.com · 17/04/2026
nice of the Barcelona Airport to make me a sign
jeff in front of sign that says, "Best Ham in the Airport"
180
Jeffrey P. Bigham @jeffreybigham.com · 06/04/2026
too many people got the brain rot! and i don't blame AI, i blame YOU
000
Jeffrey P. Bigham @jeffreybigham.com · 25/03/2026
big masculine energy
010
Jeffrey P. Bigham @jeffreybigham.com · 23/03/2026
wow, they are apparently wearing masks even at airports --
020
Jeffrey P. Bigham @jeffreybigham.com · 21/03/2026
they say it's impossible to cross party lines these days. but chuck norris does it. fitting. RIP
010
Jeffrey P. Bigham @jeffreybigham.com · 08/03/2026
redoing my CV and such, so far a fun experience with Claude Code
Update Todos

Discover local PDFs from cs.cmu.edu/~jbigham

Look up DOIs for publications missing them

Add publisher links (ACM DL, etc.)

Update BibTeX with enriched metadata

Update templates to show new links

Rebuild and verify all formats
271
Jeffrey P. Bigham @jeffreybigham.com · 07/03/2026
www.post-gazette.com/business/tec...
As of Friday, only seven companies are certified to operate “Highly Autonomous Vehicles” with a driver on board in Pennsylvania: Aurora, Motional, Stack, CMU, Perrone Robotics, Mapless AI and Waymo.
010
Jeffrey P. Bigham @jeffreybigham.com · 02/03/2026
hey, what did i do?
There isd a problem with your Amazon Mechanical Turk account
this account has been suspended by the Amazon Mechanical Turk team
040
Jeffrey P. Bigham @jeffreybigham.com · 19/01/2026
this photo, taken 18 years ago today in the Canary Islands for IUI 2008, is still my profile picture in a lot of places 😂
160
Jeffrey P. Bigham @jeffreybigham.com · 15/01/2026
i reference this sticker vending machine from the local ice cream shop surprisingly often -- it provides rotating 22-set curations of "butts on things" providing this photograph in case it might be useful to you in your life
butts on things sticker vending machin, shows burrito with a butt, beaker with a butt, watermelon butt, etc.
110
Jeffrey P. Bigham @jeffreybigham.com · 13/01/2026
when your wife is friends with a romance novelist and you get an amazon alert about delivery of their latest book … lol
Out for delivery: Arriving today 11am-1pm: "The Lust Crusade" and
000
Jeffrey P. Bigham @jeffreybigham.com · 24/12/2025
honestly, they seem pretty good. they often seem more "accessible" (approachable) than my own abstracts, usually seem accurate. it's related to how tweet threads about papers are always better/more compelling than abstracts, b/c of weird publication norms/expectations.
Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app’s full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.# Summary

This research addresses the long-standing problem of inaccessible mobile applications by developing an automated system to generate accessibility metadata directly from app screenshots. Many iOS and Android apps lack sufficient semantic information needed for accessibility features like screen readers to function properly, leaving people with visual impairments unable to use them effectively.

The researchers created a comprehensive dataset of 77,637 annotated screens from 4,068 iPhone apps to train their system. They collected screenshots and UI element information, then had workers manually annotate visual UI elements with detailed classifications and attributes. This dataset revealed significant accessibility gaps: 59 percent of screens had UI elements not exposed to accessibility services, and 94 percent of apps contained at least one such screen.

The core of their approach is an on-device object detection model trained to identify UI elements directly from pixels. They selected a Single Shot MultiBox Detector architecture using MobileNetV1 to balance accuracy with performance constraints, achieving 71.3 percent mean average precision on their test set. The model uses only 20MB of memory and requires just 10 milliseconds for inference, making it practical for real-time use on iPhones.

Beyond basic detection, the researchers implemented several augmentations to provide complete accessibility metadata. They added heuristics for grouping related elements, determining navigation order, and detecting UI states like selected or unselected toggles. They integrated iOS built-in features for optical character recognition, icon recognition, and image descriptions to provide content information. They also developed a separate model to predict whether icons are clickable with high precision.

To evaluate the approach, they conducted user studies with nine experienced screen reader users who tested inaccessible apps using both standard VoiceOver and their Screen Recognition system. Results showed significant improvement: usability ratings averaged 2.08 for regular VoiceOver compared to 3.73 with Screen Recognition, a statistically significant difference. Participants reported becoming aware of previously hidden UI elements and successfully using apps they had previously found completely unusable.

The system called Screen Recognition generates this metadata at runtime and makes it available to iOS VoiceOver, enabling accessibility even for apps without developer-provided metadata. While limitations remain, such as difficulty understanding complex gestures without visual affordances and occasional performance lag on complex screens, the approach demonstrates a viable pathway to dramatically improve accessibility for existing mobile applications without requiring developer cooperation. The researchers plan to extend this work to other platforms and contexts beyond mobile, and to integrate it into developer tools to help create more accessible apps from the start.AI-Generated Summary (Experimental)
This summary was generated using automated tools and was not authored or reviewed by the article's author(s). It is provided to support discovery, help readers assess relevance, and assist readers from adjacent research areas in understanding the work. It is intended to complement the author-supplied abstract, which remains the primary summary of the paper. The full article remains the authoritative version of record. Click here to learn more.

The summary is generated through a controlled process that uses a third-party service operating under contractual restrictions. Article content is used only for the purpose of generating the summary and is not used for model training, retained for reuse, or incorporated into other systems.

Click the blue "Feedback" button on the right side of this page to comment on the accuracy, clarity, and usefulness of this summary. Doing so will help inform refinements and future regenerated versions.
110
Jeffrey P. Bigham @jeffreybigham.com · 10/12/2025
voicemail transcription, any idea what it originally said?
The receiver doesn't work the ghetto
220
Jeffrey P. Bigham @jeffreybigham.com · 12/11/2025
"lockerroom talk"
010
Jeffrey P. Bigham @jeffreybigham.com · 08/11/2025
it happened, the vizwiz challenge surpassed the original vizwiz paper by citations --
Vizwiz grand challenge: Answering visual questions from blind people
D Gurari, Q Li, AJ Stangl, A Guo, C Lin, K Grauman, J Luo, JP Bigham
Proceedings of the IEEE conference on computer vision and pattern …
1193	2018
Vizwiz: nearly real-time answers to visual questions
JP Bigham, C Jayant, H Ji, G Little, A Miller, RC Miller, R Miller, ...
Proceedings of the 23nd annual ACM symposium on User interface software and …
1153	2010
060
Jeffrey P. Bigham @jeffreybigham.com · 30/10/2025
in 2014, I bought 3 nest thermostats for my house, which cost like $600… 10 years later, google apparently can’t afford to keep the api running, but had offered me an option to pay $250 per thermostat that will do exactly the same thing as my old one #internetofshit
2110
Jeffrey P. Bigham @jeffreybigham.com · 29/10/2025
i don't think my plane is gonna fit in this tiny little airport? what is that, an airport for ants?
visuazliation of my flight approaching denver, which for some reason has the plane an enormous size (bigger than all of denver)
050
Jeffrey P. Bigham @jeffreybigham.com · 07/10/2025
some beautiful fall weather for pittsburgh, which is showing off into the second week of october --
temperatures for the next five days range from 42 low to 67 high, sunny every day!
010
Jeffrey P. Bigham @jeffreybigham.com · 29/09/2025
#3 paper at #uist2025, by @mlam.bsky.social from her Apple internship, on Policy Maps, which is a visualization tool to help map out the unbounded space of LLM behaviors Wednesday, 11:24am in Miami www.cs.cmu.edu/~jbigham/pub...
Policy maps chart LLM policy coverage over an unbounded space of model behaviors. Here, an AI practitioner is
designing a policy for how an LLM should summarize violent text. Policy map abstractions (right) allow the policy designer to
interactively author and test policies that govern a model’s behavior using if-then rules over concepts. The designer can create
any desired concept by providing a simple text definition to capture cases of model behavior. Our Policy Projector tool (center)
renders cases, concepts, and policies as visual map layers to aid iterative policy design.
060
Jeffrey P. Bigham @jeffreybigham.com · 29/09/2025
#2 paper, paper by my student @yihaopeng.bsky.social, on Morae which is a web agent that is accessible to screen reader users, and also pauses when there are choices to be made see a demo tonight/Monday, and the talk Wed 2:42pm, Sydney www.cs.cmu.edu/~jbigham/pub...
 Morae is an accessible user interface agent that proactively pauses automation at key decision points for blind and
low-vision users to make choices. For example, when asked to “buy the cheapest sweetened sparkling water”, existing agents
would automatically choose a product even if multiple items have identical prices. Instead, Morae pauses automation and
presents relevant product differences (e.g., flavors, ratings) and allows users to choose based on their preferences. By proactively
detecting and pausing when user input is necessary, Morae allow users to actively express preferences during UI automation.
140
Jeffrey P. Bigham @jeffreybigham.com · 29/09/2025
I have three papers at #uist2025 -- #1. StepWrite, how can we compose long-form text with speech -- works better than chatGPT and dictation Tuesday, 2:54pm, Capri room www.cs.cmu.edu/~jbigham/pub...
example of hands busy scenario, writing an email, by answering targeted questions generated by the LLM-powered system
140
Jeffrey P. Bigham @jeffreybigham.com · 29/09/2025
i'm at #uist2025 this week in beautiful Busan, Korea -- yesterday, i got to serve on the Doctoral Consortium panel, amazing to see the great work of 10 rising star students in technical HCI … and look at this amazing lunch! :)
an amazing bento box
1200
Jeffrey P. Bigham @jeffreybigham.com · 08/09/2025
recognizing customer demand, or at least fake internet outrage, my research group has reverted our logo to the original. regardless of our logo, we have always remained committed to delicious homestyle HCI systems and accessibility research like my grandma used to make.
cracker barrel style logo with a guy that might kind of look like Jeff bigham sitting next to a beaker and a logo that says, "Big Lab"
1110
Jeffrey P. Bigham @jeffreybigham.com · 06/09/2025
gotta get some sunglasses
020
Jeffrey P. Bigham @jeffreybigham.com · 25/08/2025
yes it is.
Business Insider
11 hours ago
Vibe coding startup CEO says a computer science degree is no longer the 'entry ticket' to a career in tech
270
Jeffrey P. Bigham @jeffreybigham.com · 10/08/2025
highway is closed today, as a warmup to next summer when it'll be closed for a whole month… google maps is directing folks through the tiny winding neighorhood/park roads, "official detour" is nowhere to be found… already a mess on a sunday, can't even imagine how bad this is gonna be!
map of recommended detours through park
021
Jeffrey P. Bigham @jeffreybigham.com · 06/08/2025
tired: genAI is fine wired: we need to build whole massive efforts to help ensure that genAI doesn’t tell people to eat rocks inspired: let’s put AI in nuclear weapons, for some reason that we’ll figure out later
170
Jeffrey P. Bigham @jeffreybigham.com · 06/08/2025
100% -- as a quite seasoned runner, including in cities, there are many places where i will avoid the intersections, as weird as that might sound. cars can be MORE distracted at intersections than they are on the straight parts of the road.
But pedestrians navigating a landscape built for cars rightly perceive intersections as unsafe, too. On West Hudson, pedestrians who have pressed the button for a walk signal still must contend with fast-moving cars turning as they cross.
220
Jeffrey P. Bigham @jeffreybigham.com · 06/08/2025
i'm all in favor of calling out where tariffs are costing us additional money, but of course businesses are using them as another way to just sneak random undisclosed fees onto the final receipt … i guess it's for the shirt (or the glass i bought), not the actual run haha
tariff surcharge: $2.00
120
Jeffrey P. Bigham @jeffreybigham.com · 01/08/2025
a few things differentiate me from JD Vance, in addition to me having more appalachian bonafides than he does, … I grew up never questioning what was possible… so, when i achieved things, it seemed normal I always thought this was the rural strength, he accepts it as a foregone weakness
Archived clip of Vance: I realized that in this new world, I was the cultural alien. I began to think seriously about questions that had nagged at me since I was a teenager: Why has no one else from my high school made it to the Ivy League? Why are people like me so poorly represented in America’s elite institutions? Why is domestic strife so common in families like mine? Why did I think that places like Yale and Harvard were so unreachable? Why did successful people feel so different?
140
Jeffrey P. Bigham @jeffreybigham.com · 12/07/2025
they should put these absurd requirements directly on the box, I don’t even have ONE capable adult…
1111
Jeffrey P. Bigham @jeffreybigham.com · 09/07/2025
new nomination for "internet of shit" -- google is withdrawing support for my Nest thermostat (that works perfectly fine), but has kindly offered me the opportunity to pay $150 for a new one that will (ideally) do the same thing =
 

Dear Nest Learning Thermostat owner,

‌

Thanks for making Nest a part of your home. We truly value you as a longtime customer. We want to make you aware of some upcoming changes that will affect our earliest thermostats, including those at Bigham House.


Starting October 25, 2025, Google will no longer support the Nest Learning Thermostat (1st gen) introduced in 2011 and Nest Learning Thermostat (2nd gen) launched in 2012. Over the years, we’ve brought out newer thermostats with even more ways to help you save energy and stay comfy. As part of our ongoing efforts to provide the best possible smart home experience, we are transitioning our thermostat infrastructure to a new platform offering significant advancements in performance, reliability, and features that will greatly benefit our customers. Unfortunately, we’re not able to move our older thermostats to this platform.


You’ll still be able to adjust the temperature, mode, schedules, and settings on your thermostat – and existing schedules should continue to work uninterrupted. However, your thermostat will no longer receive software or security updates, will not have any Nest app or Google Home app controls, and will not support other connected features like Home/Away Assist. See more details at our support website.


To help with this transition and to show our appreciation for your loyalty, we’re offering you the option to upgrade to our newest and most advanced thermostat, the Nest Learning Thermostat (4th gen), at a special price: $149.99 (MSRP $279.99).1 Please click here to redeem the offer through the Google Store (online only). Offer valid until November 21, 2025 (if you’ve already redeemed your offer, no further action is required).
080
Jeffrey P. Bigham @jeffreybigham.com · 06/07/2025
not entirely sure why i keep doing this to myself, but figured I could probably act like an AC… lol … also @barik.net promised me I wouldn't have to sit on a 6 hour PC meeting phone call where I have no opportunity to do anything and nothing is actually decided. didn't you Titus?
Jeffrey Bigham -

Thank you for agreeing to act as a committee member for CHI 2026 Papers.
260
Jeffrey P. Bigham @jeffreybigham.com · 21/06/2025
Come on Carol! (adventures in old signs)
carol never wore her safety goggles, now she doesn’t need them
020
Jeffrey P. Bigham @jeffreybigham.com · 05/06/2025
ok, actually, this is much scarier …
251
Jeffrey P. Bigham @jeffreybigham.com · 05/06/2025
ChatGPT is getting pretty good! Although, now we’re back to copy pasting probably, but I love how it guides them to describe what about the paper they like
130
Jeffrey P. Bigham @jeffreybigham.com · 20/05/2025
i voted!
Jeff with "I voted sticker"
070
Jeffrey P. Bigham @jeffreybigham.com · 19/05/2025
(somewhat random) nytimes op-ed linked to @hcii.cmu.edu's "Human-AI Interaction" course that i started long ago, as an example of the kind of things we should be teaching and, sure, it's not a "bachelor program", but an individual course. www.nytimes.com/2025/05/19/o...
First, we need to ensure workers are learning the skills employers are starting to demand. New approaches are emerging: American University’s Kogod School of Business is embedding A.I. across its curriculum and training faculty members to use A.I. tools, while Carnegie Mellon is offering an A.I. bachelor’s program in which students take classes on harnessing the power of A.I. to be “beneficial and useful for people.” Community colleges in Miami-Dade, Houston and Maricopa Community Colleges are joining in, starting a national A.I. consortium to align curriculums with work force needs and offer applied A.I. degrees backed by companies like Intel and Microsoft.
080
Jeffrey P. Bigham @jeffreybigham.com · 11/05/2025
90s evening fundraiser with my lady
280