Authentication added.

New firmware version 1.3.9 has been just released. It includes few minor improvements, and as main new feature an authentication, that was added to web interface.  How it works is pictured on right side. Every new client connection is asked for user and password that must match with internal values. Default user is: admin, and default password is: pass. Both then can be changed in Settings.

At first I was thinking to use obsolete HTTP basic authentication, to make it simple. But it is very obsolete, and also not supported by vanilla HTTPD from lwip contrib package. I was thinking how to add an authentication for a longer time. There are several methods to be used. First would be creating a single page application with authentication message encrypted in URL as parameters. Witch would require a bigger re-work on the HTML page generation. Now HTTPD has several pages, on which user can set tens of options for each section on a page. Quite a difficult task.  Second, to use a cookie to store some secret after user authenticate successfully on browser side, and request the browser to send this secret with each request. But lwip's HTTPD does not support cookies. 

Well I deiced for second option. I created a simple patch to HTTPD  to handle cookie generation and reception. First is done as part of dynamic header option for pages that are not statically liked, that is not generated via makefsdata program. Then, I added an additional callback, that is preceding request for every fs_open (file system open function). This allows an application to verify received cookie and then, if needed, rewrite the page( or file) requested to something else, for example login page.

I think it is nice and neat addition to lwip's HTTPD, so I will try to push it upstream, hoping it will get accepted.

New pre-compiled version is as always available GitHub, together with all sources. Enjoy!

P.S. Finally this is new version for my "production" home gateway. Resetting it after 163 days. Quite satisfied 💪

Forum migration

 Nabble forum was migrated to new server http://ohs.356.s1.nabble.com

Thanks to Nabble for support and keeping this small forum up and free. 

Summer vacation and 1.3.8 continuity test

Summer vacation is almost over, I hope for everyone it was spent in nice and restful manner. I personally, was traveling abroad and also much in my country to visit places I've wanted to see for longer time. After long stay-at-home period of covid 19, it was very welcome and pleasant.

But as always, there is the the other side of vacation, I did not do do much development for OHS. The last one which bothered me about lwip's MQTT was solved, and it took longer time. Also as I think about it now, the current firmware has all the major functionality I wanted initially. And it is in state that it simply works. Yes! Look at the picture, I have now running the last 83 days without interruption.

So what will come next? I have few ideas:

• There is a big task ahead to integrate SSL/TLS layer.
  
• Based on above create "push" notification as new alert delivery path.
• Add authentication to web interface.
  
• Create examples for OHS integrated scripting language (modified TCL).
• ...

If you have some other idea let me know.

 

Solving lwip troubles.

Some time after the introduction of lwip core locking into ChibiOS port, I started to notice that MQTT still tends to disconnect and connect right after. Same problem as before core locking. This is not very good that all other lwip process seems to work well, but MQTT is keep on reconnecting. Well lwip is a big and complex library to jump in and decode all possible problems, and I decided to start with MQTT client part only. There are quite a few debugging facilities in lwip and I turned on many to start seeing the whole picture.

Nice informative way to see information about lwip is to enable LWIP_STATS. There are many options what to see, like to examine possible memory problems by defining:
#define LWIP_STATS_DISPLAY 1
#define MEMP_STATS 1
#define PBUF_STATS 1
#define SYS_STATS 1

And adding:

#ifdef LWIP_STATS
  stats_display(); // Show lwip statistics
#endif

into while (true) lwip thread.

I started to torture again OHS application fir frequent radio packets resulting into MQTT publish messages. All was fine even with heavy usage of web interface and MQTT subscribe. But after 5 days the statistics started to show an error on MEMP TCP PCB and the in used number of same did not cease even after no other TCP communication then MQTT was used. I increased the number of MEMP_NUM_TCP_PCB to 50 and started again. 5 days later the same problem.

Examining MQTT library or searching forums did not help much. In logs I found that the client receives an empty pbuf which is forcing the client to disconnect. Mosquitto broker then does not see the keep alive periodic packet and also closes the connection. But I started to measure the exact time when the MQTT get out of pbufs, and it a bit surprised me that it happens exactly after 4.9 days. To be precise after 2^32 / 10000 = 429496,7296 seconds. There are timers in lwip that make various time dependent tasks, like removing used pbufs or dhcp. Deep diving into it I found out that the problem is not in lwip, but in ChibiOS port. The sys_now() function does not overflow on u32_t boundary but actually much earlier, causing the timers wait forever.

A patch of the sys_now() function seems to fix the problem completely. It comes with new OHS version 1.3.8 with a few cosmetics enhancements described in particular commits on GitHub. 

It is recommended to upgrade if you use MQTT.

Fixing MQTT

There was lately a lot of changes done in gateway MQTT processing. First of all I've introduced a lwip_assert_core_locked() function which present as helper macro throughout lwip network stack, but it is defined as empty. This macro is a quite useful feature, especially if lwip is running multiple sockets. ChibiOS has very functional and well defined bindings to lwip, however lacks the lwip_assert_core_locked() function definition. To benefit from this lock I've added following code to lwipopts.h:

#if LWIP_TCPIP_CORE_LOCKING
void lwip_assert_core_locked(void);
#define LWIP_ASSERT_CORE_LOCKED() lwip_assert_core_locked();
#endif

And the actual function to /arch/sys_arch.c:

#if LWIP_TCPIP_CORE_LOCKING
void lwip_assert_core_locked(void) {
  // If the mutex hasn't been initialized yet, then give it a pass.
  if (lock_tcpip_core == SYS_SEM_NULL) return;
  chSysLock();
  // Ensure that the mutex is currently taken (locked).
  if (chSemWaitTimeoutS(lock_tcpip_core, TIME_IMMEDIATE) == MSG_OK) {
    chSysHalt("TCPIP core is not locked!");
  }
  chSysUnlock();
}
#endif

Then basically you need to wrap every lwip API call with following macros:
LOCK_TCPIP_CORE();
lwip_function();
UNLOCK_TCPIP_CORE();


This helped a lot, since without the locking the gateway after some time was not sending ACK to MQTT broker, and was forced to reconnect every 5 or so minutes.

Second lwip source was updated to latest dev branch, which has some fixes for MQTT in it as well. 

Last, but not least, there are several changes introduced to MQTT thread. The connection/re-connection function was moved there as well from service thread.

All this is is pushed to GitHub as version 1.3.6.

Release notification

Just a short post about OHS firmware. In case you would like to be notified about new version being  available for download, you can subscribe in GitHub directly by "Watch" button on top right. Either watch all, or "custom" and then just the releases.

More about releases, I've just pushed few commits to GitHub releasing 1.3.5. It's adding more support for remote zones, adding debug message to alarm thread, and fixing few small bugs.

Authentication node changes

As you maybe know, the OHS gateway 2.x offers except standard arm away mode also arm home mode.
This allows gateway to monitor only zones, that you manually setup out of specific group, while in arm home state. This change required also some changes in authentication mechanism. As authentication nodes I offer does not have any keyboard to differentiate between arm home and arm away state, they do it based on time the iButton is connected to the probe. That is, short touch and the group receive an arm away command, while long (over 1 second) touch of the probe, and group receive arm home command. 

This working quite well, but after using it for several months I found out that the disarm command also waits for the end of scanning period (1.4 second), trying to recognize arm home / away command. Which had me wondering I put the iButton correctly many times while disarming. Then I realized it does not have to wait for it at all. Simply there is no need, as the disarm command does not differ as arm state. Luckily the node already knows which state the group is in. Realizing all that, I've updated the authentication node sketches and put new version on GitHub. Now the scanning period has two different time values, 1.4 second for arm state, and 0.4 second for disarm state. It just feels more comfortable to disarm if you receive instant feedback.

While on it, I've also updated the set default function, to make it more clear what and where is all set. And then, I've used these changes to propagate values of the elements. Hopefully making it simpler to understand. Feel free to update.

One more small hardware change, I've changed the iButton probe connector and matting PCB connector to JST X2.54, it is much flatter then previously used connector allowing it to fit in narrower spaces.

Zone expansion board

Over last few weeks I was playing with new addition to OHS, zone expansion board. The
board has 8 analog(balanced) and 1 digital(unbalanced) ports. Analog ports are hybrid, meaning that they can be switched in software to unbalanced. Connection can be established via wire(RS485) or radio(RFM69). It has onboard relay and similar power detection circuit as gateway. There are 2 free pins left that are taken out to allow some user application like additional relay board.

The relay present on expansion board can added as basic relay output, or it can be registered as remote Horn/Siren, by the configuration definition described in expansion board sketch. Also new functionality was added into gateway firmware, which is now able to recognize such remote Horn/Siren nodes and turn them on, when associated group is in alarm state.

Sketch is, as always, present in GitHub. And it is also available for sale.

2.0.4 gateway footprint

In case someone would like to drill holes to hold the gateway up in metal casing, or have better understanding of connectors and components on 2.0.4 PCB I've put a PDF file in Google drive for print.

A7600E modem tests

The new SimCom A7600E modem is pin to pin compatible and working well in the gateway, so far no software changes needed.  Supporting both 2G and 4G/LTE. One benefit is that it comes up and also shuts down faster then 7600E or G.

2G:

4G/LTE:

MQTT subscribe

A new 1.3.1 version of gateway firmware is committed to GitHub which is enabling MQTT subscribe functionality.

Gateway subscribe on topics: /OHS/set/#, particularly parsers following 

/group /{#} - index of group {1 .. ALARM_GROUPS}
  /state - allowed commands {disarm, arm_home, arm_away}
/sensor - allowed only for Input nodes
 /{address} - node address like W:2:I:D:1
  /value - float value send to remote node
 

To describe it better for someone new to MQTT.  MQTT is an open OASIS and ISO standard,  lightweight, publish-subscribe network protocol that transports messages between devices. The protocol usually runs over TCP/IP. It is designed for connections with remote locations where a "small code footprint" is required or the network bandwidth is limited.

For example:

/OHS/set/group/#/state topics is able to perform disarm, arm_home, arm_away commands for group index specified as #.

/OHS/set/sensor/{address}/value topics is able to receive float number that is then passed to appropriate node address.

Above subscribe and already existing publish functionality allow direct integration with mobile applications,  logging platforms for sensor data, or Home Assistant:

• MQTT Alarm panel
  
• MQTT Sensor
  

New board revison, bulk buy, discount on 2.0.4 gateway

Last week I've received new 2.0.4 gateway boards. There are a few visible changes like connector description on silk screen and cut out hole for zip tie to hold on the RFM69 pigtail cable. And a few not so visible changes in routing and such, but none of them are really a reason to call it a new revision.

One problem that I've started to realize is relatively pricier GSM modem. The 2G/3G/4G SIM7600x are starting at 40EUR for regional and 50EUR for Global band coverage. I was looking for some cheaper alternative while still keeping at least 4G, and the apparent replacement seemed to be a new SimCom A7670x. Unfortunately these use different package, and are only covering Europe and China, still I deiced to give it a go and spun a new version of gateway. Just few days before sending it to fab I received a newsletter from SimCom that they released same modem module called A7600E which is pin compatible to one that gateway already use (7600X). One difference it is still Europe only and only 2G/4G. I think it is reasonable as 3G is, at least in Europe, being replaced by 4G/5G networks. Good thing is it requires only one antenna and it is 5EUR cheaper. I've ordered 2 units to test them.

For gateway, I have consolidated all the components, and I was able to get the price down by making bulk buy for all pricier components. Like buying the STM32F437 at RS Components, were it is 50% of my local distributor price.. This should reduce the final price to some 92EUR without a modem.