json-camera: A Photograph Stored as Text

Every photo you have ever saved went through a codec: a piece of software that decides which parts of a picture are worth keeping and which can be quietly dropped. JPEG is the famous one, and the rules it uses were written by hand in 1992 by people reasoning about how human eyes work. json-camera does not use hand written rules. It learned its own, by being shown 800 photographs and scored on the answer.

The scoring is the whole idea. Most compression research trains a network to make pictures look good and then hopes a general purpose compressor does well on whatever comes out. This one is trained against a loss with two terms fighting each other: how wrong the rebuilt picture is, and how many bits it actually cost to write down. Because the second term is a real, differentiable measure of file size, the network is optimising the thing you care about directly rather than a proxy for it. A single knob decides who wins that fight, and that knob is the only difference between a small blurry file and a large sharp one.

The result is a text file. A photograph goes in, four layers of convolution shrink it to a small grid of whole numbers, a second network works out exactly what each of those numbers costs in bits, a range coder packs them to within 1.6 percent of the theoretical limit, and the whole thing is written into readable JSON. You can open the file in a text editor and see the header. A third network reads it back and rebuilds the photograph.

On held out photographs it had never seen, at about a third of a bit per pixel, it beat JPEG on all twelve test images: 2.21 dB better on pixel accuracy and 3.46 dB better on the perceptual measure, at a matched file size. Matched is the important word. JPEG quality is binary searched until it lands on the same number of bytes, so neither codec gets to pick a flattering operating point.

Three things are worth saying plainly, because they are easy to oversell. The model weights are part of the file format, so a file can only be opened by the exact network that wrote it, and every file carries a fingerprint so a mismatch is refused rather than handed back as noise. It is compression and not encryption: the payload looks unreadable but there is no key, and anyone with the model can read it. And JSON itself costs about 25 percent over the raw bits, because text can only carry so much per character, which is a price the format pays knowingly rather than a flaw in the codec.

It also has a second mode that uses no network at all. The learned codec earns its ratio by discarding detail, and sometimes that is not acceptable. When nothing may be thrown away the job stops being about learning and becomes pure prediction: guess each pixel from the ones already decoded, and record only where the guess was wrong. A reversible colour transform, the median edge predictor, and the same range coder get a photograph back bit for bit, about 20 percent under PNG. The awkward part is that decoding sounds strictly sequential, since every pixel needs its neighbours rebuilt first. But the predictor only ever looks left and up, so a whole diagonal of the image resolves at once, which turns three million sequential steps into about three thousand.

I built the whole stack: the networks, the range coder, both file formats, a training pipeline with a proper held out validation split, a live dashboard for watching a run, and the website you use it on.