From “butt music” to MIDI: turning hidden notation in images into sound

You know that feeling in a museum when you finally stop sprinting gallery to gallery. A detail grabs you, and then another one. Suddenly the artwork stops being a “famous thing you’ve seen” and becomes a world you can enter.

That’s the technical version of what people call “thickness” in art: value that increases the more attention you bring.

A wildly memorable example is Hieronymus Bosch’s The Garden of Earthly Delights. On the hell panel, one figure appears to have musical notation painted across his butt. In 2014, Oklahoma Christian University student Amelia Hamrick transcribed what she saw into modern notation and recorded a short piece; her blog post went viral, and the melody became a punchline you can still hear as actual music.

So how do you turn scribbles in a centuries-old painting into something a computer (or a musician) can play? That question lands right in the middle of computer vision and music transcription.

“Thickness” as a signal-extraction problem

When you look at a dense artwork, you’re doing active decoding: finding where the “useful signal” hides inside confusing surroundings.

In image processing, that same instinct shows up as a pipeline:

• Take an image.
• Isolate the region that contains the meaningful pattern.
• Enhance it so the pattern stands out.
• Convert geometry (where things are) into symbols (what they are).
• Convert symbols into a representation you can play or analyze.

The hard part is that the world isn’t cooperating. In Bosch’s case, the “notes” are tiny, painted, partially distorted by the scene, and possibly ambiguous.

That’s why thickness is tricky for beginners: the rewarding method isn’t “look once and extract.” It’s “look again, with tools, and with assumptions that you later stress-test.”

The data we’re trying to decode: notation, staff, and pitch

Before touching code, it helps to name the pieces.

Staff and note positions

Most Western music notation is built on a staff: a set of horizontal lines (usually five) where note symbols sit. Where a note appears vertically on the staff strongly hints at pitch.

Pitch without a clef is ambiguous

A clef (like treble or bass in modern notation) tells you which staff line corresponds to which pitch. Without a clef, vertical positions alone can map to multiple pitch sets.

Hamrick reportedly approached this by making a specific assumption about the mapping between staff lines and pitch (for example, treating the second line as C, which can be common in certain chant contexts). That’s a classic move in transcription: pick a plausible mapping, generate sound, then check whether it “behaves” musically.

A practical goal: produce MIDI

MIDI is a standard format for describing musical events (notes and timing) so you can render them with a synthesizer or into sheet music.

Our technical aim is to go from an image crop to a sequence of note events.

Pipeline: from image pixels to note events

Think of the steps as swapping out your eyes for a sequence of specialized detectors.

Step 1: Crop to the notation region

You rarely want to process a whole museum-worthy painting. Crop down to the area containing the staff and noteheads.

In practice, this can be manual (best for a first build), because the first success is usually “get the crop right,” not “fully automate everything.”

Step 2: Correct perspective and scale

Paintings aren’t photographed like flat documents. The staff might be at an angle.

A perspective correction (also called a homography in computer vision) transforms the image so the staff becomes approximately horizontal and evenly scaled.

You can do a quick version by selecting four corner points of the staff plane and using a projective transform.

Step 3: Enhance contrast and binarize

Notation is “dark shapes on lighter background” more often than not, but lighting and paint texture create noise.

A common move:

• Convert to grayscale.
• Enhance local contrast (for example, using CLAHE, Contrast Limited Adaptive Histogram Equalization).
• Threshold to get a black/white image that’s easier to analyze.

Step 4: Detect staff lines

We need the staff lines before we can map note y-positions to pitch.

A standard approach uses the Hough transform, which is a technique for finding lines in edge maps.

Conceptually:

1. Find edges (with something like Canny edge detection).
2. Search for strong horizontal lines.
3. Estimate the line spacing.

Even when the staff is messy, the “horizontal-ness” often survives.

Step 5: Detect noteheads (or at least note blobs)

Once staff lines are known, you can detect connected components—groups of pixels that form shapes.

For Gregorian-style square notes, noteheads might look like thick rectangles, but in a painting they can blur into nearby shapes. That means you’ll likely filter by:

• Area (too small is noise; too big is probably something else)
• Bounding box aspect ratio (noteheads have characteristic proportions)
• Distance to the staff (notes should sit near the staff grid)

Step 6: Convert y-position to pitch using an assumed mapping

Now the transcription logic starts.

If you have five staff lines, you can index them (top to bottom) and map each possible note position to a pitch.

Hamrick’s assumption about which staff line equals C is the kind of thing this step needs. In code, this becomes a mapping from “staff line index + offset” to MIDI note numbers.

Because clefs and chant conventions can vary, the mapping step should be treated as an adjustable parameter.

Step 7: Convert pitch sequence to MIDI

Finally, the detected notes become a time-ordered list. If you don’t know rhythm yet, you can start with a constant duration—enough to audition pitch correctness.

Later you can improve rhythm by measuring spacing and correlating with typical chant timing patterns.

Minimal “audition” code: staff detection + note y-mapping

This isn’t a fully automatic miracle machine. It’s a first build that turns an image crop into a MIDI guess.

pip install opencv-python numpy matplotlib music21

import cv2
import numpy as np
from music21 import stream, note

# Placeholder: replace with your own crop of the notation region.
img = cv2.imread("bosch_butt_crop.jpg")

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# 1) Contrast enhancement
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
enhanced = clahe.apply(gray)

# 2) Binarize (tune thresholding if needed)
_, bw = cv2.threshold(enhanced, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)

# 3) Edge detection for line finding
edges = cv2.Canny(bw, 50, 150, apertureSize=3)

# 4) Hough lines to find mostly-horizontal staff lines
lines = cv2.HoughLinesP(
 edges, 1, np.pi/180,
 threshold=80,
 minLineLength=200,
 maxLineGap=10
)

horizontal_ys = []
if lines is not None:
 for x1, y1, x2, y2 in lines[:,0]:
 # Keep near-horizontal lines
 if abs(y2 - y1) < 5:
 horizontal_ys.append((y1 + y2)/2)

# Cluster staff lines by y coordinate
horizontal_ys = np.array(horizontal_ys)
if len(horizontal_ys) < 3:
 raise RuntimeError("Not enough staff-line candidates. Try a better crop or parameters.")

horizontal_ys.sort()

# Simple clustering: group by proximity
clusters = []
for y in horizontal_ys:
 if not clusters or abs(y - clusters[-1][-1]) > 8:
 clusters.append([y])
 else:
 clusters[-1].append(y)

staff_lines = [np.mean(c) for c in clusters]
# Keep the five best lines (for typical notation)
staff_lines = sorted(staff_lines)[:5]

# 5) Detect candidate note blobs (connected components)
num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(bw, connectivity=8)

candidates = []
for i in range(1, num_labels):
 x, y, w, h, area = stats[i]
 # Filter by rough size constraints
 if area < 80 or area > 5000:
 continue

 # Candidate y-center
 cy = centroids[i][1]
 # Notes should sit near the staff grid; allow some slack
 if cy < min(staff_lines) - 40 or cy > max(staff_lines) + 40:
 continue

 candidates.append((cy, area, (x,y,w,h)))

# Deduplicate nearby candidates (notes may be split/merged)
candidates.sort(key=lambda t: t[0])
filtered = []
for cy, area, bb in candidates:
 if not filtered or abs(cy - filtered[-1][0]) > 10:
 filtered.append((cy, area, bb))

# 6) Map y position to staff-step index
# Assume staff_lines are evenly spaced and note positions align to half-steps.
# This is where Hamrick-style assumptions enter.

staff_lines = np.array(staff_lines)
line_spacing = np.median(np.diff(staff_lines))

# Map: staff line index -> MIDI pitch (example mapping)
# Treat bottom line as E4 and go upwards in steps of diatonic scale degrees.
# You’ll likely need to tune this mapping for the specific notation.

def y_to_step(y):
 # Convert y to a normalized grid coordinate
 # Higher on image = lower numeric y
 # Step 0 at bottom staff line
 bottom = staff_lines.max()
 rel = bottom - y
 return rel / (line_spacing/2) # half-line steps

# Convert step to MIDI (very rough audition mapping)
# You can replace this with a chant-aware mapping later.
step_to_midi = {
 0: 64, # bottom line
 1: 65,
 2: 67,
 3: 69,
 4: 71,
 5: 72,
 -1: 62,
}

# Build a music21 stream
s = stream.Stream()
for cy, area, bb in filtered:
 step = y_to_step(cy)
 # Round to nearest integer step
 k = int(np.round(step))
 if k not in step_to_midi:
 continue
 n = note.Note(step_to_midi[k])
 n.duration.quarterLength = 1.0
 s.append(n)

s.show("midi")

What this code does well:

• It demonstrates the core logic: detect staff lines, detect note blobs, map y → pitch.
• It creates an audible “audition MIDI” quickly.

What it won’t do on its own:

• Perfect automatic pitch mapping when clefs/notation conventions are unclear.
• Rhythm extraction from static painted spacing.
• Robust detection across wildly different images.

That’s the “thickness” lesson: the first pass gets you close enough to understand the ambiguity, and then you iterate.

Why this approach matches Hamrick’s story

Hamrick’s transcribing moment wasn’t a computer vision breakthrough. It was attention plus a music-theory choice about how to interpret a staff.

In technical terms, she likely ran an implicit version of the pipeline above:

• isolate the region (the butt notation)
• interpret it as staff-like markings
• assume a pitch mapping
• render a sound
• let the result tell her whether the assumptions make musical sense

In both art and engineering, the “treasure room” isn’t reached by one heroic guess. It’s reached by repeatedly entering the cave with better lights.

Closing note: thickness becomes engineering when you can measure it

Once we translate “this artwork rewards attention” into an extraction pipeline, thickness stops being a vibe and becomes a method: isolate signal, define assumptions, and test.

That’s why hidden notations in paintings still matter. They’re weird, dense data. They force us to build tools that don’t just see pixels—they make reasoned symbols from them.

And honestly, that’s the same reason good teachers and good books can “thicken” you in the first place: they train you to stay long enough to earn the meaning behind the surface.