Publish Spec First, Think Later

Aaaargh!

MQTT is a messaging specification for a lightweight messaging protocol for the internet of things.

It has this gem in it:

Section 2.3.1

… SUBSCRIBE, UNSUBSCRIBE, and PUBLISH (in cases where QoS > 0) Control Packets MUST contain a non-zero 16-bit Packet Identifier [MQTT-2.3.1-1]. Each time a Client sends a new packet of one of these types it MUST assign it a currently unused Packet Identifier [MQTT-2.3.1-2]

OK, so what happens  65,536 packets later when we run out of unique identifiers?

Uh, hey, guys and gals, you do understand if this is a 10packet/sec high feed from some instrument (this IS, after all, an IoT messaging spec) in just under 2 hours you’re going to run out of unique identifiers?

Then there’s this gem from section 3.1

The payload contains one or more encoded fields. They specify a unique Client identifier for the Client, a Will topic, Will Message, User Name and Password. All but the Client identifier are optional and their presence is determined based on flags in the variable header.

Okey dokey… and how, in the name of Ba’al the Soul Eater, Death, Destroyer of Worlds, King Of Kings, is the client supposed to obtain a unique client identifier?

And since this identifier is not the User Name, does this mean any user can pretend to be any client?

::Annie is here, sacrificing a chicken to Bumba, the god who vomits forth the world::

 

 

Posted in Uncategorized | Leave a comment

The Snail

I’m building a robotic snail named Pomatia.

They’s a research tool to examine how robots and children interact. I’ll take them (her? him? Snails are simultaneous hermaphrodites) around to libraries and schools and interact with children.

I have a long term project to have one robot for each learner in K3 classrooms.

She’s about 60 cm long, and has a plush outer covering.

Internally she’s powered by a variety of DC motors and a vacuum pump.

She has a ‘jamming gripper’ underneath to pick things up, and a receipt printer so she can leave behind scraps of paper with notes on them, and a speaker to talk.

She’s equipped with an array of sensors – camera, touch sensors, acoustic ranging sensor, passive infrared sensor, microphone, edge-of-table IR reflective sensor.

She can move various parts of her body. She has bulldozer like treads that move her.

Powering all of this is a large NiMH battery.  She can recharge by moving atop a charging station.

Much of her computation is off board, on a conventional laptop. The laptop acts as a wireless connection for the robot.

She works in conjunction with some stand off sensors – notably a kinect. This does some of the SLAM work. She’s more comfortable in a room with fiducials.

She’s semi-autonomous. High level behaviors are triggered by a human minder using a semi-covert presentation controller of the type used by sales people. More complex control is done through a localhost browser based interface in the laptop.

Yes, this is all crazy.

 

 

 

Posted in Uncategorized | Leave a comment

Actively avoiding knowing what you’re doing

Some years ago I needed 256channels of analog output for a hack, and was scared to attach my home brew gizmo to my expensive computer (desktops were expensive then). Trying to keep costs low, I looked outta the box.

My solution was to epoxy photoresistors to the face of an old monitor, and trigger them by painting squares on the monitor. One power darlington + 1 photocell per channel, low part count.

…. I don’t actually recommend doing this, though it did work just fine ….

 

 

Posted in Uncategorized | Leave a comment

Useful Dependencies

libraries, utilities, devops glitter, etc….

I’m not a fan of adding to the tech stack without seriously thinking about whether this trip is necessary.

Suppose we need to write FoobyCalc’s file format, .foob

Bob the engineer says it’ll take him T days to write code to write .foob format. But there’s a utility, NiftyVert, that converts .blarg, which we already write, to .foob

Of course we all have to install NiftyVert, but that’s no problem. We have e engineers, and production, and test servers, n = e + 2, and it takes time t to install it.

So, T >> nt, we’re saving time….

BUT… we now have to install NiftyVert on every system, including all future hires, and Nancy’s machine after she spills her latte in the old one, we have to deal with new versions of NiftyVert, we have to keep fiddling with NiftyVert, a process that takes a small fraction of each engineer’s time, say k, where k is 0.01 or something.

So, if L is the development lifetime of the software, we invest Lnk in NiftyVert, as opposed to T in our own converter. k had better be darn small or Lnk is going to be larger than T.

And it gets worse when we do this over and over. Suppose NiftyVert’s no better or worse than anything else we install, and we install m things like NiftyVert. Now we have every engineer working at 1 –  mk efficiency, and time lost to sysadmin stuff is Lnmk.

Now, if we double the size of the project, things get even uglier.

Let’s ignore the mythical man month and say our team doubles in size. A bigger project probably also lives twice as long. n has roughly doubled (you probably now have a demo install for marketing, and one for the graphic artists to use). L has doubled because we’ve doubled the project lifetime. m has doubled, since the density of these things is constant in the size of the codebase. k hopefully goes down a little as you have more resources to make installs and such easier.

But still, at some point this rolling snowball is not only going to far exceed mT, but it can grow til it consumes most of the project’s resources. I’ve worked on projects, many of them, where it seemed impossible to get all the dependencies working, and mk was > .5

I’ve worked on a couple where mk was approximately 1.

Of course if you roll your own you’re going to find yourself extending it. So you can find yourself recoding Excel. At that point NiftyVert begins to look pretty Nifty. So keep a weather eye on Bob’s little converter he wrote, lest it become a department. If these decisions were easy, anyone could make them. Sadly, it seems no one actually makes them.

OK, everybody in the audience, raise your hands if you’ve worked on a project where it took more than a day to get a dev environment up when you were hired.

Now raise your hands if you’ve spent more than two hours dealing with something not working when it had nothing to do with what you actually were changing.

OK, everybody in the audience raise your right hand and say “I <your name> hereby promise to not shove random dependencies into the code”

 

 

 

 

 

 

 

 

Posted in Uncategorized | Leave a comment

Why Clear Standards Are Important

On 1 July 2002 a Tu154 passenger jet flown by Bashkirian Airlines, flight 2937, and a DHL cargo flight, 611, collided over Germany.

As the planes approached each other at the same altitude two systems, both designed to prevent such an occurance, cancelled each other out.

The German air traffic controller observed the imminent collision, and ordered 2937 to descend and DHX611 to climb.

Meanwhile, the automated TCAS collision avoidance system aboard both planes had triggered a verbal alarm. The system was designed to hold an election so one plane was ordered to climb, the other descend. It chose 2937 to climb, and DHX611 to descend.

The German flight manual clearly states that in the event of a conflict between TCAS and air traffic control, follow TCAS.

The Russian flight manual states that in the event of a conflict between TCAS and air traffic control, follow air traffic control.

Both crews were professional, thoroughly trained, and carried out the proper action, by the book.

72 dead. No survivors[1].

Description of the accident. The description of how the conflicting statements got in the manuals should be required reading for anyone on a standards committee.

 

[1] One more death is attributable to the crash. The flight controller was later murdered by the grieving husband and father of a woman and two children on 2937.

 

Posted in Uncategorized | Leave a comment

The Problem Domain

I started out at Apple as a contractor, working on what today we’d call ‘skins’ – being able to select the appearance, and perhaps behavior, of the borders of windows.

At the time I was learning Hindi, and struggling to improve my Hindi grammar. Now Hindi has a really regular grammar, and I thought I could write a Hindi grammar checker. If I could write a Hindi text editor with a grammar and spelling checker, it would give me a really tight feedback loop, and I hoped, my Hindi would improve rapidly.

I didn’t know much about displaying non Roman text. This was before Unicode was a thing.  Well, it was only a thing to Lee Collins and a couple other guys.  For the rest of the world, it was pretty complicated to display non-Roman text, Apple was better than anybody else at it, and and Apple Text and International were the folks who did it.

By incredible good luck, the Text and International group was adjacent to my own Muse group in Infinite Loop 1, so one day I wandered over there to ask about displaying Hindi.  By even better luck, the random person I asked was Lee. By yet more random good luck, Lee was working on the Indian Language Kit at that moment. It hadn’t gone alpha or even been completed, but Lee was willing to give me access to the repository.

Fortunately I got it built with minimal help, and started building my text editor.

Like every engineer at Apple, I had access to Radar, our bug tracker. So whenever I found a bug in the ILK, I’d report it.

After a couple months, Lee appeared at my office door one day. It seems I’d reported four times as many bugs as their two full time testers had during the period. Lee was impressed, and was there to offer me a position in text and international.  TIG didn’t have an opening for an engineer, but they did for a tester, so for a few months I was a tester, til the requisition for an engineer came through.

In that time I found lots more bugs in the ILK. I don’t believe I’m some super tester. I’m just a working programmer. The very skilled testers around me weren’t finding nearly as many. And the kind of bugs they were finding were different. They found bugs like ‘prints garbage on such and such a model printer when characters are really big’.  I found bugs in how the characters combined (in Hindi, when you have two consonants adjacent, you draw a single character that’s a combination of the two).

The difference? I knew Hindi!

So, the requisition came through, and I moved on to engineering tasks. They replaced me in the testing slot with a guy whose first language was Bangla. And hey, he found a lot more bugs in the Bangla language support than I had.

Knowing the problem domain is important.

One last story about designing and not knowing the problem domain.

I had this nice long sleeve top with a blue batik print of a Chinese character. One day our mandarin speaker laughed at my shirt.  Apparently there are two words for welcome in Mandarin, an informal one, for saying to someone as greeting, and a formal one that mostly gets used on a hanging thing next to one’s door, sorta like a welcome mat in the US.  Apparently my shirt was copied from the welcome mat one.

What you see depends on what you know.

At Electronic Arts they used to have an annual art show. Any employee could submit work.

It was a funny event – a mix of outsider art from the engineers and tech writers, and professional art from the artists.

I attended with a friend, an art director at EA.  We were wandering around, and I asked her what she liked best. She picked out a composition of black balls and lines on a clear acetate background, mounted in a frame.

I got a good laugh when I read the name on it. It was the head of our hardware department. The piece was ‘art’ alright – it was the printed circuit board photomask for the cartridge simulator – referred to by electrical engineers as ‘art’.

 

 

 

Posted in Uncategorized | Leave a comment

Transpiling

Bob Nystrom is, apparently, writing a book, and tweeted an image for it that analogizes compilation to mountain climbing routes.

Made me think about some of the more Rococo compilation pipelines I’ve set up.

Self documenting code:

Run SWI-Prolog application that pulls selected modules out of other prolog applications, searches their code for library API calls, and does other introspection tricks, and reads the structured comments annotations, to assemble this info:

  • Tree structure of available HTTP endpoints, with parameters
  • Command line options
  • Pengine publics not from system

Combines this with bits of markdown describing various parts of the system, and serves as the public API’s for the system.

After all, why use structured comments and literate programming when you can use the actual code to document?

Probably this trick only works if you can truly make sense of code programmatically. I’ve only really been able to do that with Prolog, despite other languages making a big deal of homoiconicity. Clojure fn bodies tend to be opaque.

Just because you make me check in Java:

Excel spreadsheet with a lot of information about computer vision building blocks.

The spreadsheet has some formulas that convert this info to Prolog code.

Export the Prolog code, run thru sed to fix up a line ending problem, execute the code, which generates a turtle file.

Run another Prolog program that reads the turtle file, generates a mass of Java files, and the XML crap that Maven, OSGi, and the rest of Java’s broken ecosystem requires.

Of Course the engineers need Photoshop, it’s in the toolchain

I’ve worked on a lot of graphics/games/entertainment stuff with cute bouncing bunnies (hence the blog name).  The graphic artists work in psd files, at high resolution. The final website or whatever needs lo res png’s.  But we make the artists do that conversion by hand each time they check in.

Dumb, dumb dumb. Have some engineer spend a few hours automating this.

Fun With Visio:

Hey, this problem would be easier to solve in a dataflow language.

Some hours of VBA later, I can set all this up by dragging symbols around in Visio.

OK, enough with the photoshop in the toolchain already

EZ way to write pick logic.

Create a CLUT (indexed color) image with whatever tool’s easy. When you want to do pick logic, just look up the color of the corresponding pixel and map that to whatever control.  The OK button is that icky shade of green.

I usually just use a normal rgb image and make a line of pixels in the UL corner with the palette, since nobody uses CLUT’s any more

Hint: You can store the image at lower resolution if you don’t care about pick accuracy.

Bug #47 fixed by correcting autodocs

Michael Hendrix’s  mavis pack adds types to SWI-Prolog. Yay!

So Prolog’s typeless. Where does the type info come from?

The pldoc literate programming system, of course!

Too much automation

A friend has things set up so running make pushes to github.

Patching Executables

Years ago on DOS we wrote the serial number for our product directly into the executable, then patched the checksum. Only took me a couple hours to write that bit of code, including locating the executable format (pre internet) and figuring it out. Sigh, things get more complex.

For what it’s worth,  I compiled with what would be the serial number string holding some unique marker like “hi, I am the marker”.  Then I searched through the executable for it. In those days it was worth persisting the offset to a file.

Yet more Photoshop in the toolchain

Writing a game?  The pick logic from image trick above also works for impassable terrain, power ups, etc.

If you do collision testing against a circular disc, make your life easy. Preprocess your maps.

Example: Say you have a bilevel map showing passable terrain, and 7 pixel radius  (always use odd numbers for that) sprites.  Map the bilevel image to grayscale, perform a radius 7 convolution with a hat kernal, threshold at 0.0001 and boom, you just have to test the center pixel at runtime.

All that math can be done in photoshop.  The convolution is also known as ‘blur’. 8cD

Excel for config files

Excel is an excellent choice for config files.

I bet your favorite language has a reader for ODS or xlsx or some spreadsheet format.

Stick config values in Excel cells.

What’s that give me?

Config files usually have lots of places where info’s ‘usually’ the same. Names of fourteen different servers, which, 99% of the time, are all the same. Some port numbers that are just 1 more than the previous number.  Etc. ad nauseum.

In excel, you can just have a formula.

Note that your reader will need to give you actual values, not formulas. 8cD Check that before betting career on this advice.

If that makes your manager and/or social worker hinky, have the Excel SS write out the XML or whatever abomination they’re making you use.

As Long as No One Knows, No One Gets Hurt

If you’re in Prolog, using anything but a file of prolog terms for config is a mortal sin. Say 10 Hail Marys and donate generously to Katolic Universitat Leuven or send Jan a nice recumbent bike.

If the pointy haired types insist that it’s ‘too complex’ for config, Use operators to make a DSL that looks like whatever brain dead config they want. Don’t tell.

Invent your own language

SWI-Prolog has plug in quasiquote support. I just resent that there’s no good editor support for inserting non-text stuff in them. Images? Naw, not interpretable as code.

Virtual Worlds

In Second Life you can add bits of UI to your viewer.

These HUDs, as they’re called, are made, like everything else, in the world. You build a 3D object in the world, add scripts to make it do what you want, and then use a special command to ‘glue’ it to your viewer window.

I once wrote a complier for the Pathways product.

Following a ‘build script’ it generated a bunch of objects and stuck scripts in them. I’d then pick the objects up manually (limitation of the environment) and put them in the compiler for the next pass. After a few passes I had finished product.

A Whole Gallery of crazy code representations

OK, I’m wandering off crazy toolchains to crazy code representations, but whatever. Dr. David Touretsky solicited people to represent the DeCSS algorithm in creative ways as part of fighting the DMCA. They’re preserved in a gallery here.

SWISH sorta allows natural code representations

The SWISH online tool for SWI-Prolog allows one to make ‘notebooks’, that meld markdown, HTML, and prolog code sections. The Prolog code is runnable, via pengines.

You can also register ‘renderers’. So, for example, the output of the N queens example is drawn as a chessboard.

I keep trying to convince Jan that there should also be renderer support for the code itself. So, for example, openings for a chess program would be seen by the programmer as a series of chessboards.

8c( I think Jan thinks I’m bonkers.

Don’t Settle

So, don’t settle for boring toolchains! Your toolchain can be a useful tool for increasing productivity and entertainment value.  Would that build utility be better as a minecraft object?

OK, back to my Labview!

 

 

 

Posted in Uncategorized | Leave a comment