From Pen Strokes to a Magical Diary: Building an AI Response on reMarkable

A diary that drinks your ink (and writes back)

Picture this: you’re scribbling a quick question on an e-ink tablet, you put the stylus down… and the ink you just wrote fades away. A moment later, your tablet “thinks” and draws a reply back in handwriting—stroke by stroke—without any keyboard or screen glow.

That’s the core illusion behind Tom Riddle’s diary recreations on the reMarkable Paper Pro. Technically, though, it’s not magic—it’s a pipeline that turns raw pen input into an image, sends it to a vision-capable large language model (LLM), streams the answer, and then renders the text back into strokes that match an ink-like handwriting animation. (github.com)

The moving parts: input → prompt image → LLM → handwriting replay

The interesting part is that the whole experience is built around latency hiding—making the tablet respond quickly enough that it feels alive.

1) Capturing the pen: evdev as “electrical fingerprint”

The tablet receives pen data through evdev (short for event device): a Linux interface that exposes hardware inputs as a stream of timestamped events. In this diary design, the pen events include pressure information (how hard you press) and the path of your stylus as it moves. (github.com)

On the surface, you can think of this stage as “recording strokes.” Under the hood, you’re building a time-ordered sequence of points (often including pressure and timing) that can later be rasterized into ink visuals.

2) “Committing” the page: strokes become an image (PNG)

When you pause—roughly “done writing”—the app commits the page: it converts your handwriting surface into a bitmap image. A bitmap is a grid of pixels; in practice, the diary uses PNG (a common lossless image format) to package the committed handwriting for the AI step. (github.com)

That PNG becomes the bridge between two worlds:
- your tablet’s drawing world, and
- the model’s vision world.

3) The oracle: a vision LLM that reads your handwriting

The “oracle” is the LLM backend. In the diary approach, the model is treated as a vision-capable chat system: it ingests the PNG image and produces a reply.

The project README describes two oracle backends:
- Option A: an OpenAI-compatible /chat/completions endpoint (so many providers can work through that interface), and
- Option B: a resident pi --mode rpc process (keeping a local model pipeline warm so each request doesn’t restart everything). (github.com)

A key UX trick is streaming: the oracle streams the reply sentence-by-sentence, so the tablet can begin writing before the model finishes the whole response. (github.com)

4) Handwriting synthesis: from pixels back to stroke-like ink

After the model returns text, the diary doesn’t just show it as typed characters. Instead, it uses an ink handwriting effect:
- the app represents handwritten strokes as paths,
- performs thinning/skeletonization so strokes become single-pixel “center lines,” and
- then traces those lines back into an animated replay.

The README explicitly calls out a pipeline that includes Zhang–Suen thinning (a classic thinning algorithm that reduces thick shapes to their skeleton). (github.com)

It also mentions Dancing Script as the reply handwriting font style. (github.com)

Two display strategies: windowed rendering vs. takeover mode

E-ink tablets have a unique constraint: you can’t treat the display like a regular laptop monitor. Updates are slow and the system display stack is carefully tuned for power and refresh behavior.

This is why the diary design supports two display backends.

Windowed mode (“qtfb”)—simpler, inside the UI

In the qtfb backend, the diary runs as an AppLoad application inside the main UI (xochitl). The idea is straightforward: draw to an offscreen buffer, then let the UI display it. (github.com)

This is the path for people who want fewer device-risk tradeoffs.

Takeover mode (“quill”)—lowest latency “instant ink”

The quill backend aims for the most “magical” feel by going further: it stops the vendor UI and drives the e-ink engine directly for lower latency. The project describes an interposition shim over the vendor waveform engine (via libqsgepaper.so), exposing a small C ABI such as quill_init/quill_buffer/quill_swap. (github.com)

That’s why the demo can show near-instant ink behavior when new text appears.

Where “time” gets faked: idle timers and stroke replay

A diary vibe depends on timing cues. The README describes an idle trigger (for example, an “idle 2.8s → commit page” behavior), which turns your pause into “submit my handwriting.” (github.com)

Then it uses replay:
- your input disappears,
- the model replies as it streams,
- and the reply is animated as if it’s being written live.

Even the “fade” illusion is consistent with the notion that the tablet is switching states: from “ink surface” to “thinking” to “reply surface.” The result is less like a chat app and more like a storybook object.

A practical mental model: treat the tablet like an image generator

A beginner-friendly way to understand the system is to think in four stages, each with a clear artifact:

1. Pen events (evdev) → stroke sequence
2. Stroke sequence → committed PNG image
3. PNG → LLM text stream (sentence-by-sentence)
4. Text stream → handwriting replay rendered on e-ink

That structure is a reusable pattern for many “AI on real input” interfaces—not just diaries.

Minimal “architecture sketch” (pseudocode)

The exact implementation is device-specific, but the orchestration pattern looks like this:

// 1) Listen for pen events (evdev) and build strokes
strokes = collect_pen_events();

// 2) When the user pauses long enough, commit to an image
if user_idle_for(duration_ms=2800) {
    png = render_strokes_to_png(strokes);

    // 3) Send to vision oracle and stream sentences
    let mut stream = oracle_chat_stream(png);

    // 4) For each sentence chunk, animate handwriting replay
    while let Some(sentence) = stream.next() {
        ink_paths = handwriting_to_paths(sentence);
        display_animate(ink_paths);
    }
}

Two details make this feel fast:
- you start rendering/animating as soon as sentence chunks arrive, and
- you keep the display update pipeline tuned to e-ink constraints (windowed vs takeover).

Performance reality: “first ink” is the UX metric

In the README’s described behavior for an API-based oracle setup, the system reports roughly ~0.9–1.1 seconds to first ink on-device after initiating a turn. (github.com)

That number matters because it aligns with human perception: if the pause-to-reply window is short enough, your brain starts treating the device like it has intent.

What “Fable turned it magical” really means

Recent coverage highlights that creators used Anthropic’s Fable 5 in demos that produce the disappearing-ink prompt and then generate a handwriting-style response. (inshorts.com)

In practice, the diary pattern doesn’t depend on one specific model. It depends on having a vision-capable chat model that can read the committed PNG handwriting and respond in a controlled persona style. The rest—streaming, skeletonization/thinning, and e-ink handwriting replay—does the heavy lifting for the “diary” illusion. (github.com)

Conclusion: the diary is a pipeline designed for feeling

The Tom Riddle diary effect on a reMarkable Paper Pro is a great example of engineering for emotion. It starts with pen capture via evdev, converts handwriting into a PNG for a vision LLM, streams text early, and then turns that text back into stroke-like ink animations using thinning/skeletonization and handwriting rendering. (github.com)

Once you see it as an input→image→LLM→ink pipeline, the “magic” becomes a set of concrete system decisions—especially about latency, display backends, and streaming behavior—that make the tablet feel alive on paper.