I read a job posting last week that wanted someone who could do applied data science on instrument data and also work close to the hardware. C or C++ somewhere in the requirements. Mass spectrometers on the other side of it.
I am a good fit for most of that. I run a time-of-flight mass spectrometer. I specified and own the acquisition system feeding it. I have spent years turning detector signals into numbers that survive peer review.
And there is a section in the middle of that chain I could not explain to you if you asked.
The question I could not answer
Here is the exercise I set myself, and I could not complete it:
Follow one photoelectron from the face of the detector to one row in my HDF5 file. Name every layer it passes through. At each layer, say what could go wrong in a way that produces a plausible but wrong number.
Not "what could crash." What could produce a number that looks fine and is not. That is the failure mode that matters in measurement, and it is the one you can only reason about if you know what is actually happening.
My honest map:
| Layer | Where I stand |
|---|---|
| Electron hits the detector, becomes an analog pulse | I own this |
| Discriminator turns the pulse into a clean digital edge | I know the concept, not the circuit |
| The TDC timestamps it, in silicon, on an FPGA | Black box |
| Peripheral registers and memory-mapped I/O | Black box |
| The bus carries it into host memory, via DMA | Black box |
| Kernel driver, interrupts, buffering | Black box |
| Vendor library | I call it. I do not understand it |
| My Python: decode, pair, calibrate, write the file | I own this |
Eight layers. I own the first and the last. The six in between, I have been trusting.
That is a much more useful statement than "I should learn C." It says exactly what is missing and exactly how much of it there is.
Why the standard advice is wrong for people like me
Every embedded tutorial I have found starts from "you have never touched hardware." Blink an LED. Here is what a GPIO pin is. Here is a breadboard.
That is not my starting point, and following it would cost me a year re-learning things I already know from the bench. I have written an RS232 driver against a power meter's ASCII command grammar. I have done motion control over VISA, including homing and backlash compensation. I understand Poisson accidentals and dead time. I know what a signal is before anything digital touches it.
The gap between me and a strong new graduate is not knowledge of abstractions. They have those, often fresher than mine.
The difference is that I have been lied to by a system and had to find out why.
Not a crash. A crash is honest. I spent three weeks on a device that accepted a valid instruction, quietly carried out a smaller version of it, and reported nothing. The data would have come out looking complete. I only found it because I eventually stopped debugging the layer I was comfortable in and went down one level, to a capability register I had never thought to read. I have written that one up separately: The Feature That Wasn't There.
That experience is the thing worth building on, and it is why my reading should be organised around determinism and silent failure rather than around "here is how a pin works."
What I am actually going to do
I wrote this out as a plan rather than a wish, because "learn embedded" is not a plan.
The machine model, once and properly. Computer Systems: A Programmer's Perspective. Not an embedded book and not a theory book; it exists to destroy the illusion that your program runs on an abstract machine. Data representation, the memory hierarchy, virtual memory, exceptions and interrupts. Then Drepper's memory paper for caches and DMA.
This is the layer that makes memory-mapped I/O stop being magic. It is also the layer that explains, directly, why a cached read of a device register can hand back a stale value and why you need memory barriers around a DMA buffer. That is not an abstract concern for me. That is the exact shape of a bug where a timestamp buffer looks fine and is not.
C, read as a hardware description language. I do not need to become a professional C programmer. I need C the way I needed that power meter's command grammar: as the notation the hardware documentation is written in. Pointers as addresses. volatile, and when it is genuinely required rather than cargo-culted. Fixed-width types. Struct layout and packing, which is the payoff: a peripheral's whole register block declared as a packed struct at a fixed address, so one line of C compiles to exactly one store to exactly one address.
The chip's own manual as the primary text. This suits how I already work. I read instrument manuals for a living, and a microcontroller reference manual is the same genre of document. The books are commentary; the datasheet is the source. GPIO, then timers, then interrupts, then DMA, then a serial bus. Timers and DMA are the ones that matter for instrument work.
Then measure it. Toggle a pin at the entry and exit of an interrupt handler, put it on the workshop scope, and watch the jitter. That is one hour, with hardware the electrical workshop already has, and I expect it to teach me more than a month of reading. Worst-case latency matters, not average. Which is the same instinct as dead-time analysis: the tail is the physics.
The part I was avoiding
There is a category I had been ignoring entirely, and it probably has the best return of anything on the list: the lab control ecosystems. EPICS, which runs large physics facilities worldwide. ARTIQ and Sinara, which run quantum labs. Tango. QCoDeS.
These are named in real job postings. I have written my own acquisition system, which means I will be asked why I did not use one of these, and right now my answer is defensive rather than informed. Reading enough EPICS or ARTIQ documentation to speak about it credibly is a matter of weeks, and it maps straight onto roles I am already looking at.
Further out, and much more expensive, is FPGA work. My TDC is an FPGA inside. Every serious TDC is, every fast coincidence system, every trigger and gate generator in physics. The people who write that gateware are a small population and they are paid accordingly. Six to twelve months of real effort, and it should not start until the C and interrupt material is solid. I have parked it as a 2027 goal, and I am writing that down partly so I do not quietly start it next month instead of doing the boring layers first.
Why write this down
Two reasons.
The first is that naming a gap precisely makes it smaller. "I should learn embedded systems" is a mood. "Layers three through seven of my own measurement chain, and here is the reading for each" is a task list. I have been carrying the first version around for a while and it never turned into anything.
The second is that this is, I think, the honest thing to say in an interview. Not that I know embedded systems, because I do not. That I own both ends of the measurement chain, I know exactly which part in the middle I have been taking on faith, and I am closing it deliberately. Someone hiring for an instrument team can evaluate that claim. They cannot evaluate enthusiasm.
It is also not a career change. I already do half of this. I am finishing it.