CYD Thermal Camera

PlatformIO firmware for the ESP32-2432S028R "Cheap Yellow Display" (CYD): an MLX90640 thermal camera viewer, plus a bring-up/calibration sketch for the board's display and resistive touch.

Hardware

• Board: ESP32-2432S028R, a 2.8" 320x240 ILI9341 SPI TFT with XPT2046 resistive touch, on an ESP32-WROOM-32 module. No PSRAM, ~330 KB usable heap.
• Sensor: Melexis MLX90640, a 32x24 pixel IR thermal array, connected via I2C to CN1 - the 4-pin connector nearest the microSD slot. That header is the only place the board exposes two free bidirectional GPIOs (27 and 22), so it's the only sensible place to hang an I2C peripheral.
• SD card: the onboard microSD slot, wired to the ESP32's other SPI bus (VSPI), so it doesn't contend with the display (which runs on HSPI).
• Shutter: the board's BOOT button (GPIO0). The ROM bootloader only reads it at power-on to decide whether to enter flashing mode, so once the sketch is running it's a free, ordinary input.

Build targets

Two firmwares share one platformio.ini, switched with -e:

pio run -e bringup -t upload -t monitor   # display + touch bring-up / calibration
pio run -e thermal -t upload -t monitor   # MLX90640 thermal camera (default)

TFT_eSPI is configured entirely through build_flags in platformio.ini (panel driver, pins, fonts, SPI speeds) rather than by editing the library's User_Setup.h, so the project stays self-contained and survives a clean .pio wipe.

The platform pin is pioarduino's fork of platform-espressif32, because the official PlatformIO registry package is stuck on the older Arduino core.

The MLX90640 driver in lib/MLX90640/ is Melexis's own API, vendored directly rather than pulled in via the Adafruit MLX90640 library. Adafruit's port makes the whole Melexis API private members of its own class, which puts subpage-level reads out of reach - and subpage-level reads are what make the fast render path in thermal.cpp possible (see below).

src/bringup.cpp - display and touch bring-up

A standalone diagnostic sketch: prints chip/flash/PSRAM info over serial, lights the onboard RGB LED, and draws a full-screen crosshair wherever the touch panel is pressed.

It exists to calibrate the resistive touch panel, since raw XPT2046 readings don't map linearly onto screen coordinates and vary panel to panel. Press each of the four red corner dots; the raw touch reading is shown on screen and echoed to serial. Take the extremes across all four presses and put them into the RAW_X_MIN/MAX and RAW_Y_MIN/MAX constants at the top of the file, then reflash - loop() uses those constants with map() and constrain() to convert raw touch coordinates into screen pixels.

The onboard RGB LED reports state at a glance: green when idle, blue while a touch is being tracked.

src/thermal.cpp - the thermal camera

How the sensor measures temperature

Each of the MLX90640's 768 pixels is a thermopile: a tiny absorber that warms slightly when infrared radiation from the scene lands on it, and produces a voltage that depends on the difference between the scene and the sensor's own die. The chip digitizes these voltages along with two internal references: a supply-voltage measurement (Vdd) and a die-temperature measurement (Ta). It does not output temperatures itself. Converting raw readings into degrees is done on the ESP32 by the Melexis driver.

That conversion depends on per-unit calibration data. Every sensor is characterized at the factory and ships with its own offsets, gains, sensitivities, temperature coefficients and a list of defective pixels in an on-chip EEPROM. At boot, sensorBegin() reads the 832-word EEPROM (MLX90640_DumpEE()) and decodes it into params (MLX90640_ExtractParameters()). Every later conversion uses those coefficients, so a paramsMLX90640 from one sensor is not valid on another.

sensorBegin() then configures the measurement:

• Chess mode (MLX90640_SetChessMode()): the two subpages form a checkerboard rather than alternating rows, so a half-updated frame shows fine-grained interleaving rather than horizontal banding.
• 18-bit ADC resolution (MLX_RES_18BIT, register value 2), the factory calibration default.
• 16 Hz subpage rate (MLX_RATE_16HZ, register value 5). One subpage is half the pixels, so this gives 8 complete frames per second. Faster rates exist, but noise per pixel rises with rate and the I2C bus has to keep up.

For each subpage, MLX90640_CalculateTo() compensates the raw pixel values for Vdd, die temperature, per-pixel offset and gain, and pixel sensitivity. The result is the infrared power each pixel received. It then solves for object temperature (To) using a Stefan-Boltzmann model of what the pixel sees. Two constants at the top of thermal.cpp feed that model.

EMISSIVITY (default 0.95)

Real surfaces emit less infrared than an ideal black body at the same temperature, and reflect some of their surroundings instead. Emissivity (ε) is the fraction they emit. What each pixel receives is approximately:

received ∝ ε·To⁴ + (1 − ε)·Tr⁴        (temperatures in kelvin)

where To is the object temperature and Tr is the temperature of the surroundings being reflected. The driver inverts this to recover To, so the emissivity you pass in determines how much of the signal is attributed to the object and how much to reflection.

0.95 is a reasonable default for most everyday targets: skin (about 0.98), paint, plastic, wood, fabric, paper, and matte or oxidized surfaces. It is one global value applied to every pixel. Low-emissivity materials read badly with it. Polished metal (roughly 0.05 to 0.3) mostly reflects its surroundings, so a hot polished pan will read close to room temperature. The standard workaround is to put a patch of matte black paint or electrical tape on the object and measure that, not to adjust EMISSIVITY for the whole scene.

If ε is set too low for the target, readings are pushed away from Tr: hot objects read hotter and cold objects read colder. If it is set too high, readings are pulled toward Tr. The error grows with the difference between the object and the room. For example, a true-blackbody target at 35 °C in a 22 °C room reads roughly 0.6 °C high at ε = 0.95 (calculated from the model above, not measured).

TA_SHIFT (default 8.0 °C)

The reflected-temperature term needs a value for Tr, which the sensor cannot measure. The usual approximation is room air temperature, and the sensor's own die temperature is the nearest thing available. The die runs warmer than the room because of the chip's own power dissipation and heat from nearby parts, so loop() estimates Tr as:

const float ta = MLX90640_GetTa(mlxFrame, &params) - TA_SHIFT;
MLX90640_CalculateTo(mlxFrame, &params, EMISSIVITY, ta, frame);

The 8 °C figure is Melexis's recommended value for a bare sensor in open air. Despite the local variable being named ta, the value it holds is Tr. CalculateTo() measures the real die temperature from the frame data itself for its own compensation, and uses this argument only for the reflection term.

As a result, TA_SHIFT matters only as much as the emissivity lets it:

• At ε = 1.0, the (1 − ε) term vanishes and TA_SHIFT has no effect.
• At ε = 0.95, reflection carries about 5% of the weight, so the full 8 °C shift moves readings near room temperature by only about 0.4 °C. A lower Tr raises the computed To.
• At low emissivity, reflection dominates, and TA_SHIFT becomes significant.

8 °C suits a sensor mounted in free air. If the sensor is enclosed, or close to the ESP32 module or the display backlight, the die will run hotter and the shift should be larger. To calibrate it, let the unit warm up for a few minutes, compare MLX90640_GetTa() against a reference thermometer in the same room, and set TA_SHIFT to the difference. Given the small weight at ε = 0.95, this is only worth doing when chasing sub-degree accuracy or measuring low-emissivity targets.

Reading the sensor

The MLX90640 streams its 32x24 array as two interleaved "subpages" - in chess (checkerboard) mode, each I2C read returns one half of a checkerboard pattern of pixels, leaving the other half untouched in the result buffer. MLX90640_CalculateTo() only writes the pixels belonging to the subpage it was just given, so frame[] accumulates a complete image across two reads. Factory-calibrated bad pixels (brokenPixels) and outlier pixels (outlierPixels) are corrected every frame via MLX90640_BadPixelsCorrection(), interpolating defect locations from valid neighbors.

The Melexis library in lib/MLX90640/ has been optimized for ESP32 hardware: emulated 64-bit double math was replaced with single-precision 32-bit hardware FPU operations (sqrtf, float literals), inner chess-pattern checks use fast bitwise masks, and I2C chunk transfers were doubled from 32 to 64 words. Boot-time EEPROM extraction reuses the static mlxFrame[] buffer, eliminating dynamic heap allocations.

I2C runs at 1 MHz once the sensor is confirmed up (conservatively probed at 400 kHz first, since the initial EEPROM dump is an 832-word read and the worst possible moment to discover a marginal bus). If reads start failing at speed, the bus falls back to 400 kHz automatically.

A power-cycle-proof recovery routine (i2cRecover) bit-bangs the bus free before every Wire.begin(), terminating with a manual STOP sequence. This matters because the ESP32 resetting (a reflash, a watchdog trip) does not power-cycle the sensor - it can be left mid-transaction holding SDA low, which a plain Wire.begin() retry cannot fix, since the master can't issue a START while SDA is stuck low.

Filtering and image enhancement

The raw thermal stream passes through three complementary stages:

1. Subpage-aware temporal filter: In SUBPAGE_RENDER 1 mode, an exponential moving average (FILTER_ALPHA) smooths sensor noise across time. Only pixels belonging to the active subpage are updated each cycle, eliminating the checkerboard phase lag and tearing of a naive full-frame filter.
2. Spatial 3×3 Gaussian pre-filter: A separable 1D Gaussian kernel [0.25, 0.50, 0.25] is applied across rows and columns with clamp-to-edge boundary replication. This attenuates single-pixel NETD noise before upscaling, preventing random sensor jitter from expanding into large blotches.
3. Thermal boundary sharpening: A discrete 2D Laplacian operator evaluates edge gradients on the Gaussian-smoothed field and applies high-pass boost (SHARPEN_BETA 0.35). Because the field is already smoothed of sensor noise, this selectively sharpens real physical boundaries (edges of people, hot components, drafts) without amplifying noise grain.

Rendering

The 32x24 sensor image is upscaled 8x to 256x192 and drawn into the left side of the 320x240 landscape display, with a colour scale bar and temperature readouts in the right-hand margin and a centre-point readout and frame rate in the bottom margin.

Key performance and image quality choices:

• Bicubic Catmull-Rom interpolation: Replaces bilinear upscaling with 4-point cubic Hermite splines (C¹ continuous tangents). Contours and isotherms become smooth, rounded curves rather than faceted diamonds.
• Precomputed mapping tables: Weight and coordinate maps (xs0..xs3, ys0..ys3, xw0..xw3, yw0..yw3) are built once at boot (buildMaps()) with horizontal/vertical mirror flipping folded in.
• Separable scanline evaluation: renderBand() computes a 32-element vertical cubic slice once per scanline (128 multiply-adds), then evaluates 4-point horizontal splines across the 256 screen pixels, about 4.5 multiply-adds per output pixel instead of 16.
• Asynchronous double-buffered SPI DMA: The image is rendered and pushed in 8-row bands (BAND), two 4 KB buffers instead of a 98 KB framebuffer. While one is being transferred over SPI via pushImageDMA(), the other is being computed (USE_DMA 1 with -O3 optimization). Computing a full image takes roughly 10 ms against roughly 18 ms of SPI at 40 MHz (estimates, not benchmarks), so the overlap hides most of the arithmetic.
• SPI wire-order colour LUT: A 256-entry Inferno palette is built once at boot with bytes pre-swapped into SPI wire order (lut[i] = (c >> 8) | (c << 8)). This allows DMA to transmit directly to the ILI9341 display with zero runtime conversion, ensuring correct hue rendering and avoiding little-endian false contours.
• Subpage rendering: The screen redraws after every subpage (SUBPAGE_RENDER 1), delivering a smooth 16 Hz display update rate matching the sensor's subpage frequency.

Auto-ranging and readouts

The colour scale's hot/cold endpoints auto-range every frame using the 2nd/98th percentiles (std::nth_element on a scratch buffer) rather than raw min/max, with NaN guards and minimum span protection (SPAN_MIN). The range itself is temporally smoothed (SMOOTH) to prevent flicker.

The crosshair temperature averages the 4 central sensor pixels of the sharpened field, ensuring both checkerboard subpages contribute equally to the readout.

Screenshots

Pressing the BOOT button saves a JPEG to the SD card (/thermal0000.jpg, incrementing, existing files are never overwritten). Screenshots read back whatever is displayed on the glass via the display's MISO line, capturing the actual rendered image, colour scale bar, and UI panels.

• Tap (< 500 ms): saves a single screenshot.
• Hold (≥ 500 ms): records a 2-second burst saving every rendered frame - useful for capturing transient thermal events.
• The JPEG encoder output buffer is right-sized to 32 KB to avoid heap fragmentation.

Panel quirks (this unit)

• Display inversion must be on (-DTFT_INVERSION_ON=1). Without INVON every colour shows as its complement: red appears cyan, black appears white. This is what previously looked like a "dead red channel". It isn't one; the glass reproduces red fine once inversion is set.
• GRAM reads must be slow (SPI_READ_FREQUENCY at 6 MHz). At 20 MHz every read framed one byte late, so read-back (and every screenshot) lost red and shifted green into blue.
• pio run -e paneltest -t upload -t monitor re-checks both: it dumps the controller registers, verifies a GRAM write/read round trip, and draws colour bars plus per-channel ramps.

Wiring summary

Signal	GPIO	Bus
TFT (ILI9341)	MISO 12, MOSI 13, SCLK 14, CS 15, DC 2, BL 21	HSPI
Touch (XPT2046)	MOSI 32, MISO 39, CLK 25, CS 33, IRQ 36	VSPI (bringup only)
MLX90640	SDA 27, SCL 22	I2C, on CN1
SD card	SCK 18, MISO 19, MOSI 23, CS 5	VSPI (thermal only)
RGB LED	R 4, G 16, B 17	common anode
LDR	GPIO 34	analog in (bringup only)
Shutter	GPIO 0	BOOT button, active low