Tales from the messagebox: Not just the happy path!

When you’re designing and implementing a solution don’t forget there’s more to it than the “happy path.”

It’s going to break.

It’s a certainty that at some point something will break. You’ll get a call from users asking why they’re not getting output, or not getting the output they expect. How will you resolve this?

Leave yourself a trail of breadcrumbs.

Have you saved logs and archives of inputs and outputs? These can be invaluable in investigation and recovery, but don’t forget to secure any sensitive data! Ideally they should be on a separate write only system. Should the main system be compromised you can trust the logs to help determine the impact.

Since BizTalk uses XML and not JSON don’t forget you can embed comments in your messages. Comments are ignored by automated processes, but a comment in an archived output explaining the choices it made can be a life saver. Don’t forget to include the inputs. If data is changed after a process runs it may be nearly impossible to understand what happened after the fact.

What you didn’t check can kill you

At one employer a dev lead I worked for never considered this. He was updating a query that added a year to a customer’s subscription ending date if they met the right qualifications during the month end processing.

He commented out the SQL WHERE clause during his testing and forgot to restore it when producing the production version. Net result was tens of thousands of subscribers received an additional year of service they weren’t qualified for.

He caught it immediately when the monthly process ran and tried to restore to the latest backup. He found out there were no backups. The database team that handled backups informed us we had the level of service that included backups, but we did NOT have the level of service that ensure the backups actually succeeded. The SQL server agent running the backup code wasn’t running for weeks so the backups we did have were significantly out of date.

How did this happen? We had processes in place.

He tested his code to ensure those who deserved another year got another year. He did NOT test the inverse. That those who did NOT deserve another year did NOT get it. He could have saved his career with that one test.

The QA department checking his work failed in the same way. They created a database of deserving clients and all of them got their addition year of service. There were no records for clients that didn’t meet the criteria. QA PASS.

Test for failures too!

Create tests for more code paths. Ideally keep these tests so other developers can run them and have better assurance they didn’t accidentally break code they didn’t change.

Hopefully now you’re forewarned you’ll not fall prey to this kind of mistake.

Happy coding!

Jay

Audio for your IoT control panel

It’s relatively simple to add sound to a web based control panel for your home. The only drawback is you’re not always looking at it, your browser may not play sounds from a window or tab not focused, etc.

A good alternative is to have a device hooked to a speaker so you can play notification sounds. The Prusa 3D printer company shared a great build article on creating your own talking ClapTrap robot from the game Borderlands (completely stupid, and nothing but fun). That project used a cheap device that plays MP3s from an SD card. Which is pretty much perfect for this application.

It’s a happy coincidence that the same card is supported in Tasmota already.

Stay tuned… I’ll give you details of my build after it gets closer to completion:

— Edit: —

I got some time to work on the case design for a small speaker. The one pictured above is nearly inaudible. Here’s the front half:

The research on the MP3 player was a bit depressing. I’ve been told there are several knock off versions floating around with no clear way, electronically or otherwise, to determine which they are. They have different behaviors so getting your project working is never going to be plug and play with this board. I’ll do a little looking to see if there’s a better one. Maybe an MP3 decoder can be put on the 8266 and used with an I2C DAC chip.

Tales from the messagebox: Shared port types are landmines

I know… In school they told you it was inefficient, and just plain wrong, to write code more than once. When you created a new orchestration you referenced the port type in another already existing orchestration. Everything compiles and it works on your machine. Sweet!

Except you just put in a landmine for yourself or your posterior… uhh uhh posterity.

So here are some scenarios to think about:

1. A couple of months later they decide that old orchestration is outdated and it can go. So you delete it and when you recompile it blows chunks. You deleted the definition you’re using in the new orchestration.
2. Some other developer gets the latest code you wrote from the repository and tries to build it. It fails to compile with undefined type errors. It works on your machine, but the same code doesn’t on his. WTF?

Scenario two has a dependency loop. Here’s the setup:

• Two orchestrations, in the same assembly, Orchestration “A” and Orchestration “B”.
• Orchestration “A” depends on something in Orchestration “B”
• Orchestration “B” can’t be compiled because it depends on something in Orchestration “A”.
• This is the simple version. You can imagine a larger chain with more than two creating a circular reference problem

It compiled on one machine because when that developer coded it all the orchestrations had already been built. Visual Studio didn’t recompile them because it didn’t need to. So after introducing the circular reference it only needed to compile the new code.

So what’s the answer here?

Don’t use shared port types. I haven’t benchmarked it, but they probably aren’t saving any compile time. They’re just definitions so they aren’t testable and you can’t benefit from others testing them either.

Want some hints how to untangle it if you’ve inherited this issue?

• Find the orchestration where the port type causing problems. You can tell if the type is defined in the file you have open (not a reference) by the text font. Local definitions are in a bolder and heavier font. Other definitions in the same namespace are shown in a lighter font.
• Right click on it and select “go to definition.” Be careful here. I’ve noticed if you have more than one copy of the same orchestration open visual studio tends to crash.
• In the properties menu change the declaration type from “public” or “internal” to “private.”
• Make sure your orchestration is in its own unique namespace. If you have duplicate type names you’ll get name hiding errors at compile time.
• Select the port definition and hit Control-C to copy it. Head over to the other Orchesration(s) and hit Control-V to paste the identical definition into that file. Feels good to be bad doesn’t it? Duplicating code! You’re such a monster. 😉

Happy untangling! I’ve made a lot of money over my career cleaning up other people’s messes.

IoT: Tasmota temperature sensors

Latest project was to ensure the freezer keeps my food frozen!

I used some of the cheap 8266 boards with built in WiFi and Tasmota again. An order of DS18B20 temperature sensors and a very little soldering got a working sensor that posts the time and temperature to MQTT. A block of HTML and some calls to the truly excellent ChartJS library got me a working graph of the results!

On my home page I use a publish / subscribe model in raw HTML and Javascript. WebSockets delivers messages to and from Node-Red lets me control the whole thing.

I’m pretty happy with most of it, with the exception of Node-Red. Those guys did a great thing creating something that’s free of corporate control, then ruined it with some truly awful decisions around security, documentation, and user transparency.

Tales from the messagebox: First steps to a more trouble free infrastructure

At some point in your career you’re going to face troubles. If you’re lucky, or good, they’re someone else’s fault. There isn’t much you can do about poor choices by other programmers or your management. There are some things you can do to minimize the ones you make though.

One pain point is deployments of new code. It’s expected that it will be smooth and trouble free. If there are thousands of details to get correct it’s a big ask.

One trivially simple change you can make is to reduce dependencies. One less thing to go wrong is always a good thing. So what’s the trivial change?

Almost every demo and example you see in BizTalk tutorials is organized the same way. It’s programmer mindset, and it’s the least reliable way. They put all the schemas in one assembly, all the pipelines in another, the maps in another, and the orchestrations in yet another. It makes sense to organize them by function right? Except… In order to make sure the code you wrote and tested resides in four files. If they aren’t deployed together (and the host instances restarted to flush the cache) you get that dreaded phone call.

There’s no technical reason why they have to be separated. The code is slower to execute and more fragile. The answer is simple and doesn’t cost you a thing. All your code should be in one assembly. Your code can’t fail because not all of it was deployed.

There are exceptions of course. Any helper code you might have created can’t go in the same assembly. I’ve tried and couldn’t make it work. It should, but it doesn’t. If you know why, or have managed this feat please let me know?

p.s. Don’t forget to add folders to your visual studio project. You can organize your code neatly and still have it in the same assembly.

Have a great day.

Tales from the messagebox: Database horror tales

In one sense it’s not the worst code I’ve ever seen. In another, it is.

The worst I recall was when a Visual Basic (6) coder created an application that implemented the real estate licensing test. After completing the test for one state they were asked to implement a second. So they opened the visual layout editor and proceeded to draw the second quiz form on top of the first one.

Imagine trying to maintain two forms jumbled on top of one another. In the form open event he made all the controls for the selected state visible, and all the others invisible. It was awful, but understandable. If you weren’t aware you could have two forms and just load the one you needed…

I think I want to give the title of worst to the guy (or girl) who created a database with primary keys that are the datetime type. It worked when it was tested with an empty database. Now I’ve got to support the thing with millions of rows and I’m not allowed to fix it. “It will cost too much.” It’s okay to let it randomly fail when you get collisions, perform badly, and pay for tech support to handle tickets though.

Small things do matter, and bad choices do come back to haunt you.

Tales from the MessageBox: SendPorts for support staff

Oh carp. The orchestration is missing bindings and you have no clue what should be connected where. Maybe the deploy script failed and didn’t restore the bindings (This has never happened to me). Now you’re going to spend hours figuring it out. If it happens you’ll remember it. Here’s a simple trick you, and the support staff, will appreciate when the fit hits the shan:

When you build your orchestrations craft a good naming policy and make sure all the ports you expose follow it.

• The port name you expose in the orchestration should match the name given to the port in the management console. If you ever have an unbound orchestration you, and the support staff, can simply set the bindings so the names match. It’s easy and you’ve certain it’s correct without opening documentation or source code.
• Never use periods as separators in port names. Your orchestration won’t compile. Periods are part of the C# language. Behind the scenes BizTalk uses templates to create logical port name objects. It will confuse the hell out of the compiler.
• Don’t use something like “send port” or “sp.” Anyone looking at the BizTalk management console will know the things listed in the “send ports” panel are, you guessed it, a send port. The same can be said for orchestrations.
• Don’t use the name of the protocol. The support staff looking at the BizTalk management console may not know what an “SMTP” or a “WCF” are, and honestly they don’t care either. It’s not useful. They need stuff like “EMail_to_Clients” or “834_Enrollment_applications.”
• BizTalk has a nice text area for documentation. Use it. In six months you won’t remember any of this stuff. It helps you, the support staff, and your cow-orkers. You might even get lunch bought for you when you meet a developer that inherited the system you have moved on from (never happened to me either).

As time permits I’ll write more Tales from the MessageBox.

Sensor fusion: Tempest Weather station + Kauf Bulb

A fall can be a seriously bad event. Permanently debilitating. At our house the stairwell is sometimes darker than I would like, and I decided to do something about it.

I recently purchased a KAUF brand bulb to try out. I picked it specifically because the manufacturer supports open software. Almost all of the internet of things manufacturers force you into their walled garden. You have no other choice but to buy from them. I don’t know about you, but I resent that.

To get back to the tech, the KAUF bulb comes preloaded with ESPHome firmware so it will plug and play with the Home Assistant open source software. I don’t use Home Assistant myself, but that’s another digression. Home Assistant is too complex, too bloated, and shows every sign of becoming more of both.

KAUF does mention the bulb can be easily flashed with Tasmota though. That’s more like it! Tasmota is a lot smaller and includes features for edge computing, and more importantly for security. It’s not a good idea to leave your computers unsecured, even if they are really small. (Cue bad horror movie here!)

We already have a Tempest home weather station by weatherflow. It regularly broadcasts the weather readings over UDP on our home network. One of the readings is the Luminance outside. Every five minutes I get a reading of how sunny and bright it is.

It occurred to me the ambient light in the stairwell is directly related to the Luminance outside. So, if I were to connect the bulb to the weather station it could brighten up the dark stairwell when it was really needed. I can adjust the color and the brightness of the bulb directly as well. So began a quest to create a smart safety light.

I use Node-Red and MQTT to control the house, so below is a function node with javascript code to implement the bulb control. If the day is partly cloudy the luminance can vary considerably, so the previous three readings are retained in the flow context memory. If any of them is too dark it turns on the light. If none are too dark the lamp is shut off. If the state of the bulb is unchanged it does not hammer the bulb with worthless requests.

Math.min() gets the lowest value in the array of stored values.

payload.obs[0][9] returns the luminance value from the tempest weather station observations.

flow.get() / .set(): gets and sets a value in flow context memory.

Here’s the function node:

// turn on safety light if too dark. Value is in LUX
const tooDark = 8000;

// get history. If not defined initialize to empty array
var luminanceArray = flow.get('luminanceArray') || [];

//node.warn('luminanceArray '+ luminanceArray);

// get state of the bulb
var currentState = Math.min(...luminanceArray) < tooDark;

// luminance Value is in LUX measured outside by tempest weather station
var luminance = Number(msg.payload.obs[0][9]);

// add current value to previous measurements
luminanceArray.push( luminance );

var nextState = Math.min(...luminanceArray) < tooDark;

// keep three newest measurements
if ( luminanceArray.length > 3 )
    luminanceArray.splice(0,1);

flow.set( 'luminanceArray', luminanceArray );

// emit a message to be sent via MQTT if the state changed
if (nextState != currentState)
{
    //node.warn('send cmnd');
    // turn on safety lamp if it's too dark
    if (nextState) {
        msg.topic = 'cmnd/tasmota_xxx/Backlog';
        msg.payload = 'Color1 #28114F0000 ; Dimmer 28 ; Power ON';
    }
    else {
        msg.topic = 'cmnd/tasmota_xxx/POWER';
        msg.payload = 'OFF';
    }
    return msg;
}

//node.warn('nothing to do');
// nothing more to do. End process here
return null;

Kauf bulb issue resolved!

I purchased a Kauf bulb through Amazon because of their support for non cloud based devices. The current LED bulbs from all the stores all over drive the LEDs at the expense of the lifetime of the bulb. I’m sure you all know this, having replaced them in your own homes and businesses. My thought was if I run them at a lowered output I’d get something that lasted much longer. It might even be cheaper to buy the expensive bulbs instead of the ones that quickly end up in the land fill.

I flashed my new device to the latest tasmota and had issues right away. The flash went fine but learning to use it took some time. The device has LEDs for RGB, cool white light, and warm white light and how to control them isn’t terribly logical.

To make a dim blue light for a dark stairwell I finally settled on this command:

backlog Color1 #28114F0000 ; Dimmer 31 ; Power0 1

That worked fine, but any attempt to turn off the light caused the device to reboot and left the light unchanged.

After a “support adventure” the resolution was to down grade the device from version 13.2 to version 12.5.

I now find myself wondering it the Shelly cool white/warm white bulbs, that had similar issues are really not a Shelly hardware issue. I’m guessing these kind of bulbs are probably all manufactured in China by the same company. I’ll try down grading a Shelly bulb to test the theory. I may owe Shelly an apology.

esp8266 clock using Tasmota

I was looking for something else, and stumbled across an interesting post:

Tasmota internet clock

Tasmota is general purpose open source firmware for cheap Wifi capable microcomputer boards. A few bucks and a little software magic later you have something useful… like a clock.

Features:

• Has NTP built in so it will automatically get the correct time
• Changes when daylight savings time does
• Adjustable brightness

The author used a D1 mini board, which I don’t have. I do have a hand full of extra esp8266 boards though. I looked up the documentation for the tasmota module:

https://github.com/tasmota/docs/blob/development/docs/TM163x.md

It documents the setup, wiring, and the commands needed to send the time to the board. It worked pretty much on the first try. I did have to hunt around for a USB cable that had the data wiring. A lot of the cheapo cables just carry power and dispensed with the other two wires.

You’ll also need the timezone table so you can set it up correctly:

https://tasmota.github.io/docs/Timezone-Table/

I need to print a case for it too, but there are quite a few free on the 3D models sites.

———— Update! —————

Finished it up with a case: 

———— Update! —————

One of the important tenets of Edge Computing is to push as much behavior off the server as possible. If the connection to the server is down the system should continue to work in a reasonable manner. I.E. if the internet is down you should still be able to turn the lights on and off.

I’d like to change the brightness down at night to save power and make it easier to sleep. So I’ll push that feature to the clock itself using Tasmota rules and timers.

Here’s the script to setup timers to change the brightness based on time of day:

(The ‘#’ denotes a comment. Don’t paste that into the tasmota console.)

# become a clock on boot up
Rule1 ON System#Boot DO DisplayClock 1 ENDON
Rule1 1

# setup timers. Normal sunrise is fine, enable timers
Sunrise 1
Timers 1

# set Timer2 to 10 minutes after sunset every day of the week, run a rule
Timer2 {"Enable":1,"Mode":2,"Time":"0:10","Window":0,"Days":"SMTWTFS","Repeat":1,"Output":1,"Action":3}

# set Timer3 to 10 minutes after sunrise every day of the week, run a rule
Timer3 {"Enable":1,"Mode":1,"Time":"0:10","Window":0,"Days":"SMTWTFS","Repeat":1,"Output":1,"Action":3}

# concatenate all the events into one rule
# turn brightness up 10 minutes after sunrise
# down 10 minutes after sunset
# down even more after 11pm ( 23 hours after midnight, 23 * 60 = 660 minutes after midnight)
Rule2
 ON Clock#Timer=2 DO DisplayDimmer 26 ENDON
 ON Clock#Timer=3 DO DisplayDimmer 44 ENDON
 ON Time#Minute=660 DO DisplayDimmer 13 ENDON

Rule2 1