Build a Raspberry Pi Vehicle Interior Monitor – Mobile Broadband
In this blog series I'm documenting my maker journey as I build a Raspberry Pi-based vehicle interior monitor (PiVIM). This is the full list of posts in the series:
- Build a Raspberry Pi Vehicle Interior Monitor—Overview
- Build a Raspberry Pi Vehicle Interior Monitor—Screen Test
- Build a Raspberry Pi Vehicle Interior Monitor—Mobile Broadband (this post)
- Build a Raspberry Pi Vehicle Interior Monitor—Temperature Monitoring
- Build a Raspberry Pi Vehicle Interior Monitor—Running Code at Startup
- Build a Raspberry Pi Vehicle Interior Monitor—Putting it All Together (coming soon)
In this post I'm configuring PiVIM with mobile broadband connectivity. At this stage I don't yet know whether I will connect to PiVIM to query its status or whether I'll have PiVIM push notifications out (for example by SMS), however either way I do know I need some sort of connectivity. Setting the Raspberry Pi up as a WiFi hotspot would be a neat solution however since I need a range of up to 1 km I ruled this option out in favour of mobile broadband.
Let's Get Physical
My first task was to choose the physical mobile broadband device. A Google search for raspberry pi mobile broadband turns up quite a few hits for Huawei mobile USB dongles and what seem to be quite a lot of configuration steps to get them to work. However a friend recommended the ZTE range of USB dongles and I ended up buying a ZTE MF730M for testing purposes. This is a 3G unit and is well under under half the cost of the ZTE MF823 4G unit, however at some point I'll upgrade to the 4G version since it's more flexible. I was prepared for a painful experience to get it working but on an updated version of the latest Raspbian Jessie the ZTE MF730M just worked in true plug and play fashion.
In order to get the ZTE MF730M working I needed a SIM. I wanted to avoid a plan where monthly credit would be lost if it weren't used, since PiVIM won't get much use in winter but will get a lot of use in summer. The Three network have a PAYG SIM which fits the bill perfectly since the credit lasts for as long as it isn't used. In the UK these can be bought from Tesco for £0.99. You'll need to install it in to the dongle and leave it to activate (somewhere it can get a signal) before registering the mobile number on the Three website and adding credit.
Mobile Broadband Status
If all I wanted to do in this project was to use my mobile broadband dongle then the good news in the plug-and-play department would make for a very short blog post. I don't just want to use the dongle though, I want to display information about its status (network type, signal strength etc) on my Display-O-Tron control panel. The ZTE MF devices incorporate a web page (accessible at http://m.home) that displays status information, as well as functionality that allows the management of a phonebook and the ability to send SMSs:
It turns out that this web page gets its data via a REST API and it's possible to tap in to this API to retrieve information programatically. It's easy to see the API being used from a browser's developer tools (on the Network tab in the Chromium version that ships with Raspbian), however the good people on this GitHub site have taken the trouble to document some of the commands and have some example code.
I used their code as starting point and created a Python class to return the status of the mobile dongle via instance attributes. You can find the code on my PiVIM GitHub site as mobile_broadband.py and there is an accompanying mobile_broadband_debug.py file that has code to put the class though its paces. The Python class minus a few docstrings is as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
class MobileBroadband: """Class for working with ZTE USB Modems""" host = '192.168.0.1' url = 'http://{}/goform/goform_get_cmd_process'.format(host) hdrs = {'Referer': 'http://{}/'.format(host)} def __init__(self): self.signalbar = "" self.network_type = "" self.network_provider = "" def get_status(self): """ Returns the current values of the attributes specified in the query string. Includes a dummy data set to ensure values are returned where it's not possible to work with the USB modem connected. """ if MobileBroadband.is_connected(): # Query string doesn't work if the cmd is split in to shorter chunks using \ query_string = {'multi_data': 1, 'isTest': 'false', 'sms_received_flag_flag': 0, 'sts_received_flag_flag': 0, 'cmd': 'modem_main_state,pin_status,loginfo,new_version_state,current_upgrade_state,is_mandatory,sms_received_flag,sts_received_flag,signalbar,network_type,network_provider,ppp_status,EX_SSID1,sta_ip_status,EX_wifi_profile,m_ssid_enable,sms_unread_num,RadioOff,simcard_roam,lan_ipaddr,station_mac,battery_charging,battery_vol_percent,battery_pers,spn_display_flag,plmn_display_flag,spn_name_data,spn_b1_flag,spn_b2_flag,realtime_tx_bytes,realtime_rx_bytes,realtime_time,realtime_tx_thrpt,realtime_rx_thrpt,monthly_rx_bytes,monthly_tx_bytes,monthly_time,date_month,data_volume_limit_switch,data_volume_limit_size,data_volume_alert_percent,data_volume_limit_unit,roam_setting_option,upg_roam_switch'} # pylint: disable=line-too-long req = requests.get(MobileBroadband.url, params=query_string, \ headers=MobileBroadband.hdrs) res = json.loads(req.text, strict=False) else: res = {"modem_main_state":"modem_init_complete", "pin_status":"0", \ "loginfo":"ok", "new_version_state":"version_idle", \ "current_upgrade_state":"fota_idle", "is_mandatory":"", \ "sms_received_flag":"", "sts_received_flag":"", \ "signalbar":"2", "network_type":"DC-HSPA+", \ "network_provider":"3", "ppp_status":"ppp_connected", \ "EX_SSID1":"", "sta_ip_status":"", "EX_wifi_profile":"", \ "m_ssid_enable":"", "sms_unread_num":"0", "sms_dev_unread_num":"0", \ "sms_sim_unread_num":"0", "RadioOff":"1", \ "simcard_roam":"Home", "lan_ipaddr":"192.168.0.1", "station_mac":"", \ "battery_charging":"", "battery_vol_percent":"", \ "battery_pers":"", "spn_display_flag":"0", "plmn_display_flag":"1", \ "spn_name_data":"0033", "spn_b1_flag":"0", \ "spn_b2_flag":"0", "realtime_tx_bytes":"240692", \ "realtime_rx_bytes":"1265438", "realtime_time":"771", \ "realtime_tx_thrpt":"69", "realtime_rx_thrpt":"69", \ "monthly_rx_bytes":"39886898", "monthly_tx_bytes":"2365084", \ "monthly_time":"14028", "date_month":"201705", \ "data_volume_limit_switch":"0", "data_volume_limit_size":"", \ "data_volume_alert_percent":"", "data_volume_limit_unit":"", \ "roam_setting_option":"off", "upg_roam_switch":"0"} self.signalbar = res["signalbar"] self.network_type = res["network_type"] self.network_provider = res["network_provider"] @staticmethod def is_connected(): """ Determines if the mobile data dongle is connected and functioning correctly by checking for the existence of the web portal. """ try: # Requests throws an exception if a site doesn't exist req = requests.head("http://m.home") # pylint: disable=unused-variable return True except requests.ConnectionError: return False |
I'm only returning three instance variables but clearly the code can be easily amended to return as many as are needed. One slightly ugly feature of my code is a hard coded response from the REST API to cater for when the mobile dongle isn't plugged in. My code should probably throw an exception if the mobile dongle isn't plugged in however when it is it's potentially using credit which I don't really want it to do for debugging purposes. So for the time being I'll live with my hack.
Screen Scraping for Remaining Credit
One piece of data that doesn't seem to be available from the REST API is the credit remaining on the SIM. In my case though it is available by logging in to the three.co.uk website with the SIM phone number and password and navigating to the Account balance page. There's no API in use on this website as far as I can tell so retrieving the actual value is down to screen scraping. Python has several libraries that can help here and I've been using requests and the BeautifulSoup class that's part of bs4. Long story short with this is that I've burned numerous hours trying to make this work and so far have drawn a blank. The problem is in authenticating properly with the Three website so that navigating to another page is successful. Although this aspect is work in progress I'm mentioning it because in a roundabout way I learned what I think are two great Python tips:
- If you find that a Python library fails to install on Windows with the standard pip command it might well be that a compilation step failed. In this case you can try downloading an already compiled version of the library from the Unofficial Windows Binaries for Python Extension Packages site. (Note that AFAIK the 32/64-bit versions relate to the version of Python you are running and not whether you are running 32 or 64-bit Windows. Unless you have gone out of your way to install 64-bit Python you're probably running the 32-bit version.) Open a command prompt where you downloaded the file and type pip install followed by the first few characters of the library. Then use tab completion to complete the library name. In using this technique pip knows to install a library from your download rather than from the Internet.
- Jupyter Notebooks are great for working with code on a ‘trial and error' basis where you want to repeatedly evaluate the output of a statement without having to run the whole program every time. For me this was working out which BeautifulSoup syntax would return the value of an HTML element that I was interested in:
In the example notebook above, once the first four code blocks have been run I can repeatedly run the fifth block until I get the correct syntax for the statement that returns the authenticity_token. It's a real time saver over working in a more traditional code editor where the whole program needs to be run each time. You can find a good guide to getting started with Jupyter Notebooks here.
Hopefully I'll have time to pick up this screen scraping challenge again in the future. Meantime, if you are in the UK and fancy a crack at this then all you need to do is buy a £0.99 123 SIM from Tesco, pop it in your phone to activate over the Three network and then register the SIM on the Three website.
Tune in next time when I turn my attention to the hot topic of temperature measurement!
Cheers -- Graham