The old Jasik debuggerfor the classic Mac OS had this cool quote in the help somewhere. Forgive me for paraphrasing:

Any halfway decent programmer can debug the problem if they can display the relevant variables in an understandable way.

Of course for most programmers these days this means an interactive debugger. Which is fine, but tends to be misused to do step-step=step boredom debugging.

Functional and logic languages win here. The SWI-Prolog debugger includes ‘r’, retry, which restarts the execution at the enclosing predicate, so you don’t have the common step-step-over-over-oh crap… phenomenon. If you do, you just hit r, step over down to the problem, and this time step in.

Advantages of state free systems.

Other people favor print statements. There’s definitely a time for print statements. They can focus attention on what’s relevant. But they can often be like sightseeing in a submarine, and it’s perishingly easy to end up sticking debugging statements everywhere. This becomes logging.

Logging as debugging tends to be most useful for figuring out who to blame after production goes down. Needing lots of logging is often a smell that your system has grown beyond your control.

But often Jasik’s right. I’ve frequently found it useful to build a little GUI tool into my program that lets me browse the important data structures. A dynamic, animated tool to display data is often only an hour or two’s work, and gets repaid many times over.

Debuggers single step. You should ask yourself if you actually need to be doing that.

I used to have a collection of short sound clips. I’d insert calls to play these as the program ran in debug mode. This was vastly nifty – you soon learned what the happy sound was, and could step over big hunks of code fairly safely as long as they made the right hooting and burping sound.  Also handy for improving performance.

You can also make a little interface that displays a pixmap. Shift the image left one, then  draw in pixels with values for colors each frame. Make a rollover that tells you variable names based on rows.

Or any other way of displaying data in a way that’s well connected to the underlying model.

Yes, you probably do have a view as part of the final product. But it’s designed to hide abstractions. Your debug UI is designed to help you understand them.

If you can read the stack depth, a moving graph of that can be informative. So can graphs of CPU time by thread. SWI-Prolog provides such graphs as a built-in thing. Query


Back in Burroughs Cobol days we used to be able to tell what pass the compiler was in by leaning on the cabinet of the hard drive – the motion was characteristic.

Some intelligence and imagination is often a better solution for debugging than a death march with the interactive debugger.



Posted in Uncategorized | Leave a comment

Publish Spec First, Think Later


MQTT is a messaging specification for a lightweight messaging protocol for the internet of things.

It has this gem in it:

Section 2.3.1

… SUBSCRIBE, UNSUBSCRIBE, and PUBLISH (in cases where QoS > 0) Control Packets MUST contain a non-zero 16-bit Packet Identifier [MQTT-2.3.1-1]. Each time a Client sends a new packet of one of these types it MUST assign it a currently unused Packet Identifier [MQTT-2.3.1-2]

OK, so what happens  65,536 packets later when we run out of unique identifiers?

Uh, hey, guys and gals, you do understand if this is a 10packet/sec high feed from some instrument (this IS, after all, an IoT messaging spec) in just under 2 hours you’re going to run out of unique identifiers?

Then there’s this gem from section 3.1

The payload contains one or more encoded fields. They specify a unique Client identifier for the Client, a Will topic, Will Message, User Name and Password. All but the Client identifier are optional and their presence is determined based on flags in the variable header.

Okey dokey… and how, in the name of Ba’al the Soul Eater, Death, Destroyer of Worlds, King Of Kings, is the client supposed to obtain a unique client identifier?

And since this identifier is not the User Name, does this mean any user can pretend to be any client?

::Annie is here, sacrificing a chicken to Bumba, the god who vomits forth the world::



Posted in Uncategorized | Leave a comment

The Snail

I’m building a robotic snail named Pomatia.

They’s a research tool to examine how robots and children interact. I’ll take them (her? him? Snails are simultaneous hermaphrodites) around to libraries and schools and interact with children.

I have a long term project to have one robot for each learner in K3 classrooms.

She’s about 60 cm long, and has a plush outer covering.

Internally she’s powered by a variety of DC motors and a vacuum pump.

She has a ‘jamming gripper’ underneath to pick things up, and a receipt printer so she can leave behind scraps of paper with notes on them, and a speaker to talk.

She’s equipped with an array of sensors – camera, touch sensors, acoustic ranging sensor, passive infrared sensor, microphone, edge-of-table IR reflective sensor.

She can move various parts of her body. She has bulldozer like treads that move her.

Powering all of this is a large NiMH battery.  She can recharge by moving atop a charging station.

Much of her computation is off board, on a conventional laptop. The laptop acts as a wireless connection for the robot.

She works in conjunction with some stand off sensors – notably a kinect. This does some of the SLAM work. She’s more comfortable in a room with fiducials.

She’s semi-autonomous. High level behaviors are triggered by a human minder using a semi-covert presentation controller of the type used by sales people. More complex control is done through a localhost browser based interface in the laptop.

Yes, this is all crazy.




Posted in Uncategorized | Leave a comment

Actively avoiding knowing what you’re doing

Some years ago I needed 256channels of analog output for a hack, and was scared to attach my home brew gizmo to my expensive computer (desktops were expensive then). Trying to keep costs low, I looked outta the box.

My solution was to epoxy photoresistors to the face of an old monitor, and trigger them by painting squares on the monitor. One power darlington + 1 photocell per channel, low part count.

…. I don’t actually recommend doing this, though it did work just fine ….



Posted in Uncategorized | Leave a comment

Useful Dependencies

libraries, utilities, devops glitter, etc….

I’m not a fan of adding to the tech stack without seriously thinking about whether this trip is necessary.

Suppose we need to write FoobyCalc’s file format, .foob

Bob the engineer says it’ll take him T days to write code to write .foob format. But there’s a utility, NiftyVert, that converts .blarg, which we already write, to .foob

Of course we all have to install NiftyVert, but that’s no problem. We have e engineers, and production, and test servers, n = e + 2, and it takes time t to install it.

So, T >> nt, we’re saving time….

BUT… we now have to install NiftyVert on every system, including all future hires, and Nancy’s machine after she spills her latte in the old one, we have to deal with new versions of NiftyVert, we have to keep fiddling with NiftyVert, a process that takes a small fraction of each engineer’s time, say k, where k is 0.01 or something.

So, if L is the development lifetime of the software, we invest Lnk in NiftyVert, as opposed to T in our own converter. k had better be darn small or Lnk is going to be larger than T.

And it gets worse when we do this over and over. Suppose NiftyVert’s no better or worse than anything else we install, and we install m things like NiftyVert. Now we have every engineer working at 1 –  mk efficiency, and time lost to sysadmin stuff is Lnmk.

Now, if we double the size of the project, things get even uglier.

Let’s ignore the mythical man month and say our team doubles in size. A bigger project probably also lives twice as long. n has roughly doubled (you probably now have a demo install for marketing, and one for the graphic artists to use). L has doubled because we’ve doubled the project lifetime. m has doubled, since the density of these things is constant in the size of the codebase. k hopefully goes down a little as you have more resources to make installs and such easier.

But still, at some point this rolling snowball is not only going to far exceed mT, but it can grow til it consumes most of the project’s resources. I’ve worked on projects, many of them, where it seemed impossible to get all the dependencies working, and mk was > .5

I’ve worked on a couple where mk was approximately 1.

Of course if you roll your own you’re going to find yourself extending it. So you can find yourself recoding Excel. At that point NiftyVert begins to look pretty Nifty. So keep a weather eye on Bob’s little converter he wrote, lest it become a department. If these decisions were easy, anyone could make them. Sadly, it seems no one actually makes them.

OK, everybody in the audience, raise your hands if you’ve worked on a project where it took more than a day to get a dev environment up when you were hired.

Now raise your hands if you’ve spent more than two hours dealing with something not working when it had nothing to do with what you actually were changing.

OK, everybody in the audience raise your right hand and say “I <your name> hereby promise to not shove random dependencies into the code”









Posted in Uncategorized | Leave a comment

Why Clear Standards Are Important

On 1 July 2002 a Tu154 passenger jet flown by Bashkirian Airlines, flight 2937, and a DHL cargo flight, 611, collided over Germany.

As the planes approached each other at the same altitude two systems, both designed to prevent such an occurance, cancelled each other out.

The German air traffic controller observed the imminent collision, and ordered 2937 to descend and DHX611 to climb.

Meanwhile, the automated TCAS collision avoidance system aboard both planes had triggered a verbal alarm. The system was designed to hold an election so one plane was ordered to climb, the other descend. It chose 2937 to climb, and DHX611 to descend.

The German flight manual clearly states that in the event of a conflict between TCAS and air traffic control, follow TCAS.

The Russian flight manual states that in the event of a conflict between TCAS and air traffic control, follow air traffic control.

Both crews were professional, thoroughly trained, and carried out the proper action, by the book.

72 dead. No survivors[1].

Description of the accident. The description of how the conflicting statements got in the manuals should be required reading for anyone on a standards committee.


[1] One more death is attributable to the crash. The flight controller was later murdered by the grieving husband and father of a woman and two children on 2937.


Posted in Uncategorized | Leave a comment

The Problem Domain

I started out at Apple as a contractor, working on what today we’d call ‘skins’ – being able to select the appearance, and perhaps behavior, of the borders of windows.

At the time I was learning Hindi, and struggling to improve my Hindi grammar. Now Hindi has a really regular grammar, and I thought I could write a Hindi grammar checker. If I could write a Hindi text editor with a grammar and spelling checker, it would give me a really tight feedback loop, and I hoped, my Hindi would improve rapidly.

I didn’t know much about displaying non Roman text. This was before Unicode was a thing.  Well, it was only a thing to Lee Collins and a couple other guys.  For the rest of the world, it was pretty complicated to display non-Roman text, Apple was better than anybody else at it, and and Apple Text and International were the folks who did it.

By incredible good luck, the Text and International group was adjacent to my own Muse group in Infinite Loop 1, so one day I wandered over there to ask about displaying Hindi.  By even better luck, the random person I asked was Lee. By yet more random good luck, Lee was working on the Indian Language Kit at that moment. It hadn’t gone alpha or even been completed, but Lee was willing to give me access to the repository.

Fortunately I got it built with minimal help, and started building my text editor.

Like every engineer at Apple, I had access to Radar, our bug tracker. So whenever I found a bug in the ILK, I’d report it.

After a couple months, Lee appeared at my office door one day. It seems I’d reported four times as many bugs as their two full time testers had during the period. Lee was impressed, and was there to offer me a position in text and international.  TIG didn’t have an opening for an engineer, but they did for a tester, so for a few months I was a tester, til the requisition for an engineer came through.

In that time I found lots more bugs in the ILK. I don’t believe I’m some super tester. I’m just a working programmer. The very skilled testers around me weren’t finding nearly as many. And the kind of bugs they were finding were different. They found bugs like ‘prints garbage on such and such a model printer when characters are really big’.  I found bugs in how the characters combined (in Hindi, when you have two consonants adjacent, you draw a single character that’s a combination of the two).

The difference? I knew Hindi!

So, the requisition came through, and I moved on to engineering tasks. They replaced me in the testing slot with a guy whose first language was Bangla. And hey, he found a lot more bugs in the Bangla language support than I had.

Knowing the problem domain is important.

One last story about designing and not knowing the problem domain.

I had this nice long sleeve top with a blue batik print of a Chinese character. One day our mandarin speaker laughed at my shirt.  Apparently there are two words for welcome in Mandarin, an informal one, for saying to someone as greeting, and a formal one that mostly gets used on a hanging thing next to one’s door, sorta like a welcome mat in the US.  Apparently my shirt was copied from the welcome mat one.

What you see depends on what you know.

At Electronic Arts they used to have an annual art show. Any employee could submit work.

It was a funny event – a mix of outsider art from the engineers and tech writers, and professional art from the artists.

I attended with a friend, an art director at EA.  We were wandering around, and I asked her what she liked best. She picked out a composition of black balls and lines on a clear acetate background, mounted in a frame.

I got a good laugh when I read the name on it. It was the head of our hardware department. The piece was ‘art’ alright – it was the printed circuit board photomask for the cartridge simulator – referred to by electrical engineers as ‘art’.




Posted in Uncategorized | Leave a comment