Reposted by @potatoannotator.bsky.socialArianna Muti @arimuti.bsky.social · 22/09/2026Our TACL paper "Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology" is finally out, only six months after acceptance! 😅🎉 direct.mit.edu/tacl/article... 1113
potatoannotator.bsky.social @potatoannotator.bsky.social · 16/09/2026The check will review the instructions and walk through the full task process so it should find inconsistencies and impossible labels. It's also good an fixing/tweaking the UI for more complex designs. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 15/09/2026You can read the full guide to building a human annotation study with Claude Code, Codex or Cursor using potato-skill and the open-source Potato annotation tool, with six example studies. www.potatoannotator.com/blog/potato-...potatoannotator.compotato-skill: Build a Human Annotation Study with Claude Codepotato-skill lets Claude Code design a human annotation study from a plain description, then build and run it as a Potato task with its own interface and checks. 000
potatoannotator.bsky.social @potatoannotator.bsky.social · 15/09/2026Once you install the skill, describe your study to your coding agent and the agent uses the skill automatically. You can also start the skill yourself by typing /potato-skill (or $potato-skill in Codex). 110
potatoannotator.bsky.social @potatoannotator.bsky.social · 15/09/2026Install in Claude Code: /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato Codex or Cursor: npx skills add davidjurgens/potato-skill --agent codex cursor 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 15/09/2026The skill produces a standard Potato config that you can reuse and share. Build on it for your next task, send it to a collaborator, or rerun it next year on new data. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 15/09/2026Even more, the skill lets you easily add Potato's advanced features like training phases, attention checks, AI assistance and more. Rather than rolling your own custom annotation interface with a coding agent, you can quickly stand up a production-grade task in chat. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 15/09/2026With answers to a few questions from the agent, you can now build complex designs with any of Potato's 60+ annotation types, for text, audio, video, agents, dialogue and more. The agent then works through the task in a browser as an annotator would, to check the task works like you expect. 200
potatoannotator.bsky.social @potatoannotator.bsky.social · 15/09/2026Agentic tools are often how we build software now, so why not annotation tasks too? The Potato development team is excited to release potato-skill, which teaches coding agents like Claude Code, Codex and Cursor to design a full annotation task backed by Potato, an open-source annotation tool. 111
potatoannotator.bsky.social @potatoannotator.bsky.social · 14/07/2026Potato 2.0 also has a paper at ACL 2026 now, if you cite it in research. Full write-up: www.potatoannotator.com/blog/potato… Notes + install: github.com/davidjurgens/potato/rele… pip install --upgrade potato-annotation==2.7.0 000
potatoannotator.bsky.social @potatoannotator.bsky.social · 14/07/2026Lots of other fun features: a model arena with Elo + DPO export, CoT process-reward labeling, cross-document events, a living-document codebook, PDF cross-page linking, per-cohort schemas, better language localization. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 14/07/2026Using Potato in research? The tool can now auto-populate a latex snippet on your annotation process and results with agreement statistics, stats, timing info, etc. No more copy-paste errors! 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 14/07/2026Better LLM judge calibration: judge-vs-human alignment, calibration, verbosity + position-swap + ECE. Now that human labels carry IRT-based confidence intervals, "the judge agrees with people" finally has error bars :) www.potatoannotator.com/blog/trust-… 110
potatoannotator.bsky.social @potatoannotator.bsky.social · 14/07/2026Lots more agent evaluation: Pull real runs from OpenTelemetry, LangGraph, CrewAI or AutoGen, then annotate them: stepwise trajectory scoring and a clickable multi-agent graph. www.potatoannotator.com/blog/evalua… 110
potatoannotator.bsky.social @potatoannotator.bsky.social · 14/07/2026We know about machine CoT, but human CoT is hard to get. Our new "Think-Aloud" asks people to speak their thought process and then give a label, which we automatically transcribe and parse on-device. www.potatoannotator.com/blog/human-… 110
potatoannotator.bsky.social @potatoannotator.bsky.social · 14/07/2026Our new "boundary lab" shows a counterfactual edit after each label is made ("would the label still hold?") to get contrast sets for free. www.potatoannotator.com/blog/qualit… 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 14/07/2026Every label now ships with error bars: a live Item Response Theory (IRT) layer reads annotator ability and item difficulty from agreement alone. Multiplayer Rooms to support norming meetings in-tool, live agreement meter. www.potatoannotator.com/blog/labels… 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 14/07/2026Potato 2.7.0 is out--our biggest release yet. New multimodal *input* support, annotator group training, better mobile experiences, and even more insight into annotator behavior. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 05/07/2026Potato will be at #ACL2026. Come swing by our poster at Demo Session F on July 7 @ 9am to say hello! aclanthology.org/2026.acl-dem...aclanthology.orgPotato 2.0: A Comprehensive Annotation Platform with AI-in-the-Loop SupportDavid Jurgens, Michael Chen, Lina Iyer. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). 2026. 041
potatoannotator.bsky.social @potatoannotator.bsky.social · 22/06/2026Housekeeping: Potato is now GPL-3.0. Full notes and install: github.com/davidjurgens/potato/rele… pip install potato-annotation 010
potatoannotator.bsky.social @potatoannotator.bsky.social · 22/06/2026A few smaller things we've wanted for a while: a three-pane trace view for ongoing eval, much faster startup (it stopped loading the heavy ML libs until you actually need them), and it runs behind a reverse proxy now. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 22/06/2026You can also fix agent trajectories step by step. Correct what the agent got wrong and Potato hands you clean SFT or DPO training data. Useful when you're trying to make the agent better, not just grade it. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 22/06/2026For agent eval: let an LLM judge label your trajectories, then have people score a blind sample to check it. You find out where the judge actually tracks human judgment and where it quietly falls apart, instead of just taking its word for it. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 22/06/2026If you do qualitative work, there's a new QDA mode. Build the codebook as you go, highlight a quote to turn it into a code, keep memos next to the data, and search across all of it. A whole team can code the same project at once without clobbering each other's work. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 22/06/2026Potato 2.6.0 is out. Two things we're happy about this round: it now handles qualitative coding the way people actually work, and it got a lot better at evaluating AI agents. Still free, still open source. Have a poke around: huggingface.co/spaces/Blablablab/po… 141
potatoannotator.bsky.social @potatoannotator.bsky.social · 03/03/2026Lots of UX improvements that make annotation faster: - Navigation bar with status badges + skip-to-unannotated - Grid-based form layouts - Resizable display panels - Conditional display logic (show/hide fields based on responses) - PDF/Word/code rendering in-browser 000
potatoannotator.bsky.social @potatoannotator.bsky.social · 03/03/2026New support for loading data from anywhere: URLs, Google Drive, Dropbox, S3, HuggingFace datasets, Google Sheets, databases Plus: 55 validated survey instruments (SUS, NASA-TLX, UMUX, OCEAN, AttrakDiff...) ready to drop into your post-study workflow. Learn more about your annotators! 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 03/03/2026New pluggable export system with 8 formats: COCO JSON (with RLE masks), YOLO, Pascal VOC, CoNLL-2003, CoNLL-U, Mask PNG, EAF (ELAN), TextGrid (Praat) 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 03/03/2026Smarter annotation workflows: - MACE annotator competence estimation: model reliability, weight aggregation - LLM-based option highlighting: subtle AI cues without biasing - Embedding-based diversity ordering: reduce fatigue from repetitive items - Video object tracking with keyframe interpolation 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 03/03/2026New annotation types in v2.2: - N-ary event annotation (triggers + arguments) - Entity linking (Wikidata, UMLS, custom KBs) - Pairwise comparison (A/B + slider) - Triage (rapid accept/reject/skip) - Coreference chains - Conversation trees - Discontinuous spans All configurable via YAML. 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 03/03/2026Potato v2.2.0 is out! The biggest release yet: 9 new annotation schemas, pluggable export system, MACE integration, 55 survey instruments, and much more. github.com/davidjurgens...github.com 182
potatoannotator.bsky.social @potatoannotator.bsky.social · 05/02/2026 Potato 2.1 is here! Our open-source annotation platform now includes - multi-field span annotation (e.g., labeling premise/hypothesis separate) - span linking for relation extraction, and - AI-assisted visual annotation with YOLO + VLLMs—bounding boxes, labels, rationales, and many more! 000
potatoannotator.bsky.social @potatoannotator.bsky.social · 29/01/2026We're looking forward to adding new features and keeping Potato supported for years to come, so please feel free to send feature requests on the github too github.com/davidjurgens...github.comGitHub - davidjurgens/potato: potato: portable text annotation toolpotato: portable text annotation tool. Contribute to davidjurgens/potato development by creating an account on GitHub. 000
potatoannotator.bsky.social @potatoannotator.bsky.social · 29/01/2026A whole dedicated website! www.potatoannotator.com - Task design wizard! - Extensive showcase of 80+ annotation tasks from real papers - So much documentation You can easily add your tasks to our showcase with a pull request here github.com/davidjurgens...potatoannotator.comPotato - AI-Powered Annotation ToolFrom zero to annotating in minutes. Configuration-first annotation with AI features. 110
potatoannotator.bsky.social @potatoannotator.bsky.social · 29/01/2026Many, many quality of life improvements - New support for Amazon Mechanical Turk! - Passwordless servers for trusted environments - Dynamic data loading—new data can be added to a running server (h/t @mariaa.bsky.social for the request) - Easier debugging - So much new documentation 110
potatoannotator.bsky.social @potatoannotator.bsky.social · 29/01/2026Improved and customized UI - The whole Potato UI has been overhauled to support mobile annotation better - New task-specific UI customization lets you easily adjust your task's specific layout without hacking the tool It's now super easy to polish your design the way you want on Potato 110
potatoannotator.bsky.social @potatoannotator.bsky.social · 29/01/2026Flexible annotation/survey flow - Annotators can go through mandatory training - Support for multiple pre/post-annotation surveys - Chain together different annotation tasks one-after-another - Expertise-base instance routing We're working on more dynamic flows and item routing too! 110
potatoannotator.bsky.social @potatoannotator.bsky.social · 29/01/2026Built in support for quality control - Attention checks (catch distracted annotators) - Gold standard tracking - Live inter-annotator agreement metrics - Training phase with feedback We have a whole new admin dashboard to keep track of annotation process in real time 120
potatoannotator.bsky.social @potatoannotator.bsky.social · 29/01/2026Potato has finally moved beyond just text! New in 2.0: - Audio annotation with segment labeling and waveform visualization - Video annotation with frame-by-frame labeling - Image region labeling One platform for text, audio, video, and images—or multimodal! 100
potatoannotator.bsky.social @potatoannotator.bsky.social · 29/01/2026AI-Assisted Annotation Potato 2.0 works with multiple LLM providers (OpenAI, Anthropic, Gemini) and local LLMs (vLLM, Ollama) to help annotators work faster: - Intelligent hints without giving answers - Keyword highlighting - Label suggestions Your annotators get guidance. You keep control. 110
potatoannotator.bsky.social @potatoannotator.bsky.social · 29/01/2026Super proud that today we're releasing Potato 2.0 - the biggest update yet to our open-source annotation platform www.potatoannotator.com 🥔 Major UI/UX improvements. Support for image, audio & video annotation. AI-powered assistance/hints/labeling. And still free 🤩 Here's some of what's new:potatoannotator.comPotato - AI-Powered Annotation ToolFrom zero to annotating in minutes. Configuration-first annotation with AI features. 3187
