Adding MQTT to gateway 2.x

MQTT was somehow last step of adding features present in gateways 1.x to gateway 2.x. Now with last version of 1.3 release, already available in GitHub releases, this come to live. For now only publish functions are implemented, but subscribe procedures and skeleton is already prepared. Beware that 1.3 release forces whole configuration to be erased! I suggest to take screenshots of each page before upgrading. I know it is not very convenient, and I will try to implement mechanisms that such future changes will be a bit elegant. That is without the need to reconfigure whole gateway again.

MQTT publish functions are now having their own thread, that is collecting queue of publish request. The thread then pushes one publish a time when system allows it.

The publish structure is a bit different, as it now more relies on given indexes and addresses rather then on names of elements. Main topis is OHS with subtopics:

 /state {On, Off} - Indicates if system is on
 /group
   /{#} - index of group
     /name
     /state {disarmed, arming, armed_home, armed_away, triggered, disarming}
 /zone
   /{#} - index of zone
     /name
     /state {OK, alarm, tamper}
 /sensor
   /{address} - node address like W:2:K:i:0
     /name
     /value

There are other small changes like Nodes tab is now showing value of authentication nodes, that is a name of person who last armed/disarmed this node. Same name is also passed to MQTT sensor value for same node type.

You can test now new 1.3 version with initial MQTT support, but there can be some rough edges.

Expansion board

Gateways 2.x have now a slightly reduced numbers of input. This is a little trade off from previous versions 1.x that have 8 analog(balanced) and 4 digital(unbalanced) ports. 2.x gateways have increased the balanced inputs to 10 ports, and also added ability to switch any of this port to be treated as unbalanced one. But there is no digital only port. Instead I started to design new extension board that will allow to add 8 extra ports. These ports will also be hybrid, meaning that they can be switched in software to unbalanced. The board is already in fabrication and should look like picture on right.

I did not yet started coding, so I hope it will be still possible with good old ATmega328P. Also not sure if this application has need for RTOS, or if it is doable only in as standard Arduino sketch.

Anyway it has possibility to expand the zones by 8, and the connection can be established via wire(RS485) or radio(RFM69). It has onboard relay and similar power detection circuit as gateway. There are only 2 free pins left that are taken out to allow some user application like additional relay board.

Gateway firmware is already able to allow 2 such boards to be connected.

Pages reorganization.

Pages menu on right are now reorganized by gateway 1.x and 2.x for better access, and some pages like compiling and firmware were grouped together. Also gateway 2.x configuration is now complete. 

Further more new static page 2.x scripts is created where I will publish summary for scripts and TCL commands along with example scripts.

Travis CI and precompiled firmware for OHS 2.x

Over last weekend I've created a Travis continuous integration for OHS 2.x gateway repository. This CI performs a test build on every commit that is pushed to it. What it does is that it verifies that the commit is buildable, and maybe more importantly, it creates new compiled firmware related to every new commit, and places it back in releases section directly in GitHub.

This means that the latest, and also historical firmware are now available for download in https://github.com/vysocan/OHS_2-gateway/releases 

Binary files .bin can be directly used for DFU - Device Firmware Upgrade.

TCL scripts

TCL scripts is a new functionality in OHS 2.x. They allow various enhancements to be added by users to the automation part of the gateway. As the base of TCL language I used the following library: https://github.com/zserge/partcl, and embed it into gateway firmware. Scripts are handled by a separate thread that has a queue that serves all incoming scripts, and passes the result back asynchronously as a call back. Scripts are using their own separated heap, provided by umm_malloc library, that performs great heap management. Embedded SPI FRAM is used as storage for scripts with 64kB of available space. I have written a simple block access library called uBS (micro block system) that allows basic read write operations.

Scripts also have their own tab in the web interface, which allows you to create and run the TCL scripts directly on the gateway. As shown on the picture, there is statistics about the heap and the storage used by scripts. as well as helper icons </> and #. The </>, when you hover over it, shows build in language helps. And the # shows all variables currently in heap with their values.

Button "Run" passes the edited script into execution queue. "Refresh" button simply refreshes the page, allowing the "Last output:" to be seen. "Save" stores the script with given name to uBS storage, and allows this scripts to be assigned to triggers and timers.

Due to performance limitations, since scripts are not primary function of the gateway, there are build in limits to maximum script execution time, and maximum script interactions.

Language syntax

Tcl script is made up of commands separated by semicolons or newline symbols. Commands in their turn are made up of words separated by whitespace. To make whitespace a part of the word one may use double quotes or braces.

An important part of the language is command substitution, when the result of a command inside square braces is returned as a part of the outer command, e.g. puts [+ 1 2] performs addition of 1 + 2 and passes the result to puts.

Currently the only data type of the language is a string. Even numbers are stored in string format and converted to numeric types when needed.

Any symbol can be part of the word, except for the following special symbols:

•     whitespace, tab - used to delimit words
•     \r, \n, semicolon or EOF - used to delimit commands
•     Braces, square brackets, dollar sign - used for substitution and grouping

 

Interpreter

Partcl interpreter is a simple structure which keeps the current environment, array of available commands and a last result value. Interpreter logic is wrapped around two functions - evaluation and substitution.

Substitution:

• If argument starts with $ - create a temporary command [set name] and evaluate it. In Tcl $foo is just a shortcut to [set foo], which returns the value of "foo" variable in the current environment.
• If argument starts with [ - evaluate what's inside the square brackets and return the result.
• If argument is a quoted string (e.g. {foo bar}) - return it as is, just without braces.
• Otherwise return the argument as is.

Evaluation:

• Iterates over each token in a list.
• Appends words into a list.
• If the command end is met (semicolor, or newline, or end-of-file - our lexer has a special token type TCMD for them) - then find a suitable command (the first word in the list) and call it.

Each command has a name, arity (how many arguments is shall take - interpreter checks it before calling the command, use zero arity for varargs) and a C function pointer that actually implements the command. 

 

Builtin commands

• "set" - assigns value to the variable (if any) and returns the current variable value.
• "subst" - does command substitution in the argument string.
• "puts" - prints argument to the tcl_stdout buffer, followed by a newline
• "proc" - creates a new command appending it to the list of current interpreter commands. That's how user-defined commands are built.
• "if" - does a simple if {cond} {then} {cond2} {then2} {else}.
• "while" - runs a while loop while {cond} {body}. One may use "break", "continue" or "return" inside the loop to contol the flow.
• Various math operations are implemented like:  !, +, -, *, /, >, >=, <, <=, ==, !=, &&, ||.
• "string" - performs simple string manipulation, like"compare" and "length".
• "clock" - allows time and date manipulation, like "seconds", "format" and "add".
  

Gateway 2.x configuration

Just started to update new page section for gateway 2.x configuration. It can be find in menu on right. It will describe basics terminology for OHS, new shell interface for OHS 2.x gateways and of course a web interface used to configure the functionality.  Currently it is work in progress, but I will try to add all sections that are missing, and finish it as primary guide for new users.

DFU - Device Firmware Upgrade

Gateways 2.s have now a new user convenient way of upgrading firmware without any external programmer. As STM32F4xx devices include built in USB DfuSe bootloader, I was wanting to implement such functionality into OHS as well. Since software version 1.2 there is an option in shell interface to boot the device to dfu mode., and then upgrade the firmware directly via USB cable.

DFU tools:

dfu-util  is a simple multi-platform command line tool that can upload binary data directly to flash.

List DFU ready devices via: dfu-util -l

Example output of gateway 2.x, most important is the 'alt=0, name="@Internal Flash  ' descriptor that we gonna use:

Found DFU: [0483:df11] ver=2200, devnum=12, cfg=1, intf=0, path="3-6", alt=3, name="@Device Feature/0xFFFF0000/01*004 e", serial="336A34763437"
Found DFU: [0483:df11] ver=2200, devnum=12, cfg=1, intf=0, path="3-6", alt=2, name="@OTP Memory /0x1FFF7800/01*512 e,01*016 e", serial="336A34763437"
Found DFU: [0483:df11] ver=2200, devnum=12, cfg=1, intf=0, path="3-6", alt=1, name="@Option Bytes  /0x1FFFC000/01*016 e/0x1FFEC000/01*016 e", serial="336A34763437"
Found DFU: [0483:df11] ver=2200, devnum=12, cfg=1, intf=0, path="3-6", alt=0, name="@Internal Flash  /0x08000000/04*016Kg,01*064Kg,07*128Kg", serial="336A34763437"


Flash the firmware via: dfu-util -a 0 --dfuse-address 0x08000000 -D ch.bin

STM32CubeProgrammer is also multi-platform tool but with graphical user interface. One downside is that you need older Java version installed to run it, but once you have it it will work nicely.

DFU howto:

1. Connect your laptop/PC via USB cable to gateway. Status LED on gateway should change to short rapid flashes.
2. Open any terminal emulation program like Putty and open serial connection to new enumerated USB device. On Linux it will be ACM0.
3. In the terminal issue command 'boot dfu'.
4. The gateway should close the USB connection, terminal will exit, or close, and the status LED will stop blinking.
5. New USB device should be enumerated as 'STMicroelectronics STM Device in DFU Mode'
6. Use any of DFU tool to list, connect and to download(in terms of to download a file to a DfuSe device) firmware into gateway.
7. Reset the gateway via onboard reset button.

 

Skeches for nodes compatible with OHS 2.x gateway

Recently I have pushed to GitHub new example sketches for the Arduino like remote nodes. These include software modifications, that are needed for the changes made in gateway 2.x software. Hardware part of nodes is still the same for both wired(RS485) and radio(RFM69) nodes.

Changes include:

• Periodical ping to gateway. OHS 2.x gateway require a ping command from every registered node at lest one in an hour, or it is removed from node list.
• Arm home. This new functionality is added to authentication mechanism on node, precisely to iButton routine. It works as usual for arm away, that is a quick touch of iButton arms away or disarms the group. When arm home state is required, user needs hold the iButton in probe for more then 1 second. Disarm is same as for arm away, a quick touch only is enough.
• Authentication key is sent to gateway, and OK / Not OK sound is result of gateway accepting the packet with key. This does not mean that the key is valid and accepted, only that the gateway has received the key. On old node software, the OK / Not OK sound was result of iButton probe reading the iButton correctly or not.
• New method for CRC signature for RS485 packets.
  

Repositories with software for nodes compatible with OHS 2.x gateway has name starting with: OHS_2-node

I will add new node examples as I build them.

OHS 2.0.4 - final version

There are two changes to final version gateway board for 2.0.4. The PCB remain the same as well as all functionality, but I decided to opt for easier software changes in future.

What I mean is that I changed the MCU on the board from STM32F407 to SMT32F437. It has 64KB more RAM, and hardware accelerated cryptographic unit allowing better support for TLS. F437 allows also a higher nominal frequency, but for now I did not use this option, and left it on 168MHz. But it can come with new firmware if needed.

Second change is to switch to LAN7842A, an updated version of previously used LAN7820A. It has some new functionality which include WOL and Cable sense, which will not be need. But it has also lower power consumption and HP Auto-MDIX support. Basically LAN7842A is recommended for any new design.

On the software part, I've been pretty busy with new version. Porting many of the features of 1.7 gateway into 2.0, and I'm quite happy with it so far. My intention is to create gateway that will not need any other resources to function. 

For example web interface use font awesome for icons, meaning you need connection to internet to see them, as this font has hundreds of kB. But thanks to fontello I managed to create subset of icons that has less then 10kB, witch perfectly fits into 1MB of flash available on MCU. So far, only remote thing that is needed is a SMTP server, in case you wish that gateway should also send email notifications.

OHS 2.0.4

Over past few months I have prepared new version of OHS gateway called simply 2.0.4. It is new iteration of my 2.0 version fixing layout defects and integrating new features. Here are the specifications:

• STM32F407 with 1GB of flash and 128KB + 64KB CCM of RAM, running at 168MHz.
• LAN8720A Ethernet PHY with 10/100Mbit.
• Two step power supply with 12V to 3V8 switching and 3V8 to 3V3 LDO.
• Battery for RTC CR2032. STM32F407 has on-board RTC.
• 10 analog (balanced) inputs for PIR, smoke, ...
• 4G SIM7600x module
• Radio module for RFM69HCW, or alternatively RFM95.
• 128KB SPI FRAM.
• RS485 for wired connections.
• 2 relays for sirens.
• 1 tamper zone for enclosure. 
• Micro USB port.
• ST-LINK V2 port.
• 3 LEDs.

The main changes are GSM part, which now sports 4G/LTE modem from SIMCOM SIM7600x, and SPI FRAM with double capacity.

SIM7600x is decision that I've been thinking about for some time. On one hand the module is quite expensive compared to 2G modules, but on the other the 2G is becoming phase out in many countries. It is quite a dilemma, but I decided to invest into module that will have long life expectancy. There are always more difficulties. One is that the hardware needs to be able to speak with 1V8 logic presented on module, which I have solved by adding MOSFET level shifters. And second, that the module is bigger, and I neede to place it on the back of PCB. I have also removed the mini SIM holder, and replaced it with smaller and more up to date micro SIM.

There are several versions of SIM7600 module. One global, most expensive, that is supposed to work in all regions. And the regional ones for Europe, North America, South America, Asia and so on.

FRAM size has doubled for the reason to store some larger data needed conveniently.  First half (64KB) is used as before to keep 4096 log entries for gateway logger. Second half will be available to several new functionalities I plan to add to gateway software. I was also tmepted to add a uSD card, but it is easier to store data on synchronous SPI FRAM, then to access uSD with file system that can be damaged or not present. Not to speak about wiring.

Other changes not so visible are RC circuits on battery and power signals of UPC. These are nicely filtering out the spikes that are sent to gateway, eliminating the need for software compensations. Without them the gateway has wrongly reported AC power OFF signal.
Also there is now a shotky diode, to add polarity protection to RTC battery.

OHS logo

I'm having fun with the new http server. Since it serves static pages quite well I have created new logo for OHS and even derived favicon out of it. It will part of login screen, and 404 page. There is still plenty of storage on MCU flash (my STM32F407 has 1MB).

I have also polished a bit of JavaScript and it will hopefully bring a bit of comfort to user interface. Namely there is enhanced enable and disable script, that can toggle various buttons, selects and such html elements on and off. Second is a simple password verification script that can verify password text in two password inputs, before it is sent to httpd.

The scripts work with element ID, and since all OHS http elements are rendered with ID, all you need to do is add:

onclick="en()" and
onclick="dis()"

to the master element. This master that triggers other elements property enabled or disabled by defining array of affected ids, such as:

var x=document.querySelectorAll(\"#xx\");
var y=document.querySelectorAll(\"#a1,#a0\");

Where x is array of disabled when true, and y is array of disabled when false.

Moving forward with OHS2

Just to post latest update about my software progress on OHS version 2. I must say I'm quite happy that I have base functional system with many main system components running on my development board. Namely:

1. RS485 with new library for Arduino. Sending command and messages with ACKs. All the bits travel back and forth nicely and all timing look good on logic analyzer.
2. GSM module work well wit accompanied new library for SMT32.
3. Logging to FRAM.
4. Relays are fine.
5. Battery backup storage in SRAM is fine along with ruties.
6. RTC, although there is now only UTC.
7. HTTPD is working with static and dynamic html pages and and also POST is functional. Web server now includes pages for system info, zones, groups, nodes, users, keys and log.

With all this I can already use authentication, zone monitoring, groups, and data log.

If someone is interested all the source is in GitHub, and also any contribution is welcome.