I spent about three weeks convinced I had a wiring problem.
The setup is a coincidence experiment. A UV laser fires at a beam of molecules a thousand times a second. Each shot can knock an electron out of a molecule and leave an ion behind, and both of them fly off to their own detector. A time-to-digital converter, a TDC, records when each one arrived, to a precision of picoseconds. If you can pair the electron and the ion that came from the same molecule, you learn something you cannot learn from either one alone.
I also needed to rotate a quarter wave-plate during the run, to change the polarization of the light from linear to circular, and I needed the data to know which angle each event belonged to. The TDC has marker inputs for exactly this: you send it a pulse on a spare pin, it stamps the data stream, and afterwards you can split the file at the marks.
So I wired it up. No markers.
I checked the cable. I checked the pin assignment against the manual. I had the motion controller toggle the pin while I watched the 5V LVTTL line on a multimeter, and I could see the pin toggling. I updated the firmware. I updated the SDK. Still no markers.
I wanted it to be the cable. A wiring fault is an afternoon: you find the bad pin, you replace it, you move on. If the fault was in the software, it meant changing how the pipeline handled the stream, and going back through the vendor SDK and their example code to work out what I had misunderstood. That is not an afternoon.
So I kept checking the cable. The explanation I preferred was also the cheapest one to be true, and I did not notice for three weeks that those were two different reasons to believe it.
The thing I should have done first
Eventually I stopped trying to make it work and asked a different question: is this device even capable of what I am asking it to do?
There is a call in the vendor's library that reports which features the unit supports. I had never used it, because why would you check whether a device has the feature that is in its manual. It returned zero. Every feature bit, off.
Then I found the second half of it. When I enabled the markers and read the setting back, the device quietly reduced my request to just an internal one-millisecond timer and dropped everything else. It did not return an error. It did not warn. It accepted the instruction and then did something smaller.
I ran it properly to be sure: every marker enabled, the controller toggling the pin twenty-four times, the data stream open the whole time. Twenty-four edges in, zero markers out.
I sent that to the manufacturer. The answer came back quickly: the marker inputs are a paid option, and our particular unit was never licensed for them. The hardware is there. The feature is not switched on.
Nobody had done anything wrong. The option simply was not on our purchase order, years before I arrived, and nothing in the software layer ever said so out loud.
What I actually got wrong
The cable was never the problem. My debugging was.
I had assumed a hierarchy that does not exist. I treated the manual as the description of the device in front of me, when it is really the description of a product family. I treated silence from the API as success. And I spent three weeks in the physical layer, where I am comfortable, because checking a cable feels like progress in a way that reading a capability register does not.
That last one is the part worth keeping. I went to the layer I understood instead of the layer where the answer was. I would guess most people do this, and I would guess most people also describe it afterwards as bad luck.
The fix in the code was small. There is now a check at connect time that reads the settings back after enabling them and says clearly, in the log, when the device did not accept what it was told. Ten minutes of work. It would have saved three weeks.
Then the actual engineering question
Knowing why it failed did not get me my angles. I still needed each event tagged with the waveplate position, and buying the licence was not something I could decide on a Thursday.
The obvious move is a second code path: one build that uses hardware markers, another that fakes them, pick whichever the lab has. I have watched that go wrong before. Two paths means one of them is always the neglected one, and it is always the neglected one that is running the night before a deadline.
What I did instead: the software stamps the angle itself, onto the same stream channel a hardware marker would have used, snapped to the device's own timebase. Everything downstream, the coincidence sorting, the polarization analysis, the file writer, does not know or care which kind of marker it got. One setting in the config chooses the source. The hardware path is unchanged, byte for byte, so if the licence ever gets bought, nothing needs rewriting.
Software-marker mode. The banner states plainly which marker source is active, because the one thing this whole episode taught me is that the software should never be quiet about what it is actually doing.
The test was a live run: 945,000 events, stepping the waveplate from 45 degrees to 135 and back. Then I checked the angle stored in each event against an independent log of what the motion controller had actually done. They agreed.
Two columns, side by side, agreeing all the way down.
That was the point where it stopped being a workaround in my head. I had gone looking for a way around a licence I could not buy, and come out with a system that does not care which kind of marker it gets. The hardware path is still there, untouched, for whoever buys the option one day. But the thing I built under pressure is the thing I should have built anyway.
Three things I would tell someone starting this
Ask the device what it can do, not the manual. Manuals describe product lines. Capability registers describe the box on your bench. If your instrument has a way to report its own configuration, read it at startup and log it, every time.
Silence is not confirmation. A call that returns without an error has not told you it worked. It has told you it did not crash. Where it matters, write the setting, read it back, and compare. This is the single cheapest habit in instrument software and almost nobody does it.
Do not fork the code to work around hardware. Push the difference down to the lowest layer you can and let everything above it stay ignorant. A configuration switch is maintainable. A parallel build is a slow leak.
None of this is specific to my experiment. It is specific to talking to hardware you did not build, which is most instrument software.
The part that stayed with me
The failure mode here was not a crash. A crash is easy; a crash tells you.
This device accepted a valid instruction, silently did less than it was asked, and reported nothing. If I had not gone looking, I would have had a data file with no angle information in it and no indication anything was missing. I would have found out weeks later, in the analysis, probably by noticing that a result made no sense.
That is the category of failure I now spend most of my design effort on. Not the systems that fall over, but the ones that keep running and quietly give you the wrong answer.
A system that crashes is asking for help. A system that quietly does less than you asked is not.
The acquisition system is called VAJRA. I specified and own it, and it runs the coincidence spectrometer in the Structural Molecular Dynamics group of Prof. Dr. Jochen Mikosch at the University of Kassel, where I work as a postdoctoral researcher. If you write software for instruments and any of this sounds familiar, I would like to hear about it via the contact page.