Idan Horowitz
All work

Intelligent Interactive Systems · Technion course project · Winter 2023/24

The Future of Typing

A concept keyboard for augmented reality that replaces the QWERTY layout with LLM word and phrase suggestions, hands-free pointing and speech input.

  • Code
  • Gemini · Whisper · MediaPipe · Streamlit

The problem

We believe a new typing method will emerge in the near future, as augmented-reality technology reaches the world. When we came across the Wall Street Journal's review of the Apple Vision Pro, the reviewer noted “...one big problem.” (see minute 1:49): typing on a floating QWERTY keyboard.

We decided to create a new concept keyboard for the AR world, solving this problem.

What we built

Instead of letters, the keyboard offers whole words and phrases to tap.

  • LLM suggestions. Gemini proposes the next words and phrases from what has been written so far, and the suggestions update with every choice.
  • Tone control. A style selector (Facebook, WhatsApp, Google, LinkedIn) and a mood slider from angry through neutral to funny steer the suggestions; changing either refreshes them.
  • Feedback. Each set of suggestions has a refresh button and thumbs-up and thumbs-down ratings.
  • Speech input. Whisper speech-to-text covers anything the suggestions miss.
  • Hands-free pointing. A webcam script built on MediaPipe moves the cursor with your nose, or with the tip of your right index finger, and clicks with a thumb-index pinch. In one version an open left palm starts the tracking and a closed fist stops it.
Screenshot of the keyboard app: word suggestions on the left, phrase suggestions on the right, refresh and feedback buttons, a text box, a mood slider from angry to funny, and a toggle between speech and pointing input
The keyboard in the Streamlit app: word suggestions (left), phrase suggestions (right), refresh and thumbs-up or thumbs-down feedback for each, the text box, the mood slider, and the toggle between speech and pointing input.

The keyboard is a Streamlit app; the pointing scripts use OpenCV, MediaPipe and PyAutoGUI.

How the suggestions work

  • At each step, Gemini Pro is asked for seven one-word continuations and five three-word continuations of the text so far, in the chosen style and mood.
  • Refreshing asks for new suggestions that exclude the ones on screen, and duplicates between the word and phrase lists are dropped.
  • If the model returns nothing, for example when a safety filter blocks a reply, the keyboard falls back to placeholders instead of breaking.

What I would test next

The prototype was built and demonstrated, not formally evaluated. A proper test would be a small user study in a headset comparing it with the standard floating QWERTY keyboard on words per minute, error rate and effort, using the built-in thumbs-up and thumbs-down ratings to measure how often suggestions are accepted.

Team

Final project for the Intelligent Interactive Systems course (Winter 2023/24 semester), built with Jacob Link.

Next projectNeighborhood Smoothing for Calibration