An AI-powered photo-to-Lego converter that runs entirely in the browser with serverless GPU inference on Modal. Upload any photo and receive a complete Lego model: TripoSR generates a 3D mesh from the single image, trimesh voxelises it to a brick grid, CIE LAB colour mapping matches each voxel to the closest real LEGO colour, and a greedy brick-packing algorithm assembles optimised bricks.
Output: an interactive Three.js 3D viewer and a downloadable 2D instruction manual PDF — a step-by-step, layer-by-layer build guide generated automatically from the brick layout.
The pipeline has five distinct stages. First: TripoSR neural mesh generation, which takes the uploaded 2D image and produces a watertight 3D mesh within seconds on a GPU. Second: trimesh voxelisation, sampling the mesh on a regular voxel grid at configurable resolution. Third: CIE LAB colour mapping — each voxel is matched to the nearest real LEGO colour using Euclidean distance in the perceptually uniform CIE LAB colour space. Fourth: greedy brick packing, converting voxels to actual LEGO bricks by prioritising larger pieces before falling back to 1×1 studs. Fifth: output rendering and PDF generation.
TripoSR is the heart of the pipeline — it converts a single photograph into a 3D mesh in seconds using a transformer architecture trained on 3D object data. Converting a 2D photograph into a plausible 3D mesh in under five seconds is a genuinely remarkable capability. The model runs on Modal's serverless GPU infrastructure, meaning there is no idle GPU cost: a cold start spins up a GPU instance, processes the image, and returns the mesh, all within the request lifecycle.
The voxelisation pipeline — taking the TripoSR output mesh and converting it to a structured voxel grid — was written with Claude. The trimesh library provides the geometric primitives; Claude wrote the sampling strategy and the data structure design that makes the subsequent colour mapping and brick-packing stages efficient. The choice of a flat numpy array with stride-based indexing, rather than a sparse dict, was Claude's recommendation — significantly faster for the dense grids that result from complex meshes.
The LEGO colour palette spans over 180 colours across standard, transparent, metallic, and glow-in-the-dark variants. Matching arbitrary RGB values from the voxelised mesh to the nearest real LEGO colour required working in CIE LAB space, where Euclidean distance correlates with human perceptual difference. Claude wrote the colour conversion pipeline and the nearest-neighbour matching function using delta-E distance, and helped curate the palette — excluding colours that are rare or prohibitively expensive to source.
The brick-packing problem — given a voxel grid, find the minimal-count tiling using standard LEGO brick sizes — is NP-hard in the general case. Claude designed and implemented a greedy approximation that works layer-by-layer from the bottom up, scanning each layer for the largest valid brick placement, marking voxels as occupied, and repeating. The result is not globally optimal but produces naturalistic builds a human could credibly construct.
The serverless deployment architecture on Modal — defining the Python function, its GPU requirements, its dependency image, and its HTTP endpoint — was designed with Claude's assistance. Claude generated the Modal decorator patterns, the async job handling for the frontend to poll, and the response serialisation that gets the mesh data back to the browser efficiently.
A full-stack pipeline spanning AI neural mesh generation, Python geometry processing, serverless GPU compute, and React 3D browser rendering.