Creating a connected object is easy! This is often what we think, or what we believe we know when we create our first DIY project with an Arduino. When I attend IT conferences and listen to others talk about their projects completed over a weekend or two, that’s the impression I get. But strangely, when I design an object, I ask myself a lot of questions, write a lot of code, not just to make a simple lamp work, but to ensure that it works simply and securely… The saying goes that the “S” in IoT stands for security (geeks will understand), and if IoT has a bad reputation, it’s because of its history (though we can also talk about MS-DOS, W3.11, W95… too), but also because there is a lack of understanding of what IoTs really are. Through a personal experience to understand how a connected bulb works, I will attempt in this article to show you part of the non-functional code used in such a simple object, whose functional purpose can be summarized as adjusting the light intensity between 0 and 100% and changing the color hue.

Disassembling

When we talk about security, it already involves accessing the object. In IoT, the largest attack surface is undoubtedly the servers, which are accessible by everyone from anywhere. Accessing the object itself is more restricted, but for my experience, I only needed to go to the store and use a few tools.

Once extracted and cleaned from the bulb, you’ll find a very simple board with a power supply and a small module that contains nothing but an ESP32. It’s actually quite impressive to think that in this everyday object, you can find a microcontroller capable of Wi-Fi, BLE, with 4MB of Flash, 512K of SRAM, and operating around 160MHz… which is at least as powerful as my desktop computer from the early ’90s.

In the image, you can clearly see the ESP32 along with the 40MHz and 32KHz oscillators, the antenna line at the top left, which connects to one of the pins, making it the antenna output (all of this probably isn’t perfectly matched). You can also distinguish a total of three pins on the board, each with two sides, meaning six contacts. Once the power supply and antenna are removed, there are only three pins left for controlling the light.

On the other side, you can find test points, which are used for factory programming, such as Reset, Boot Configuration, Serial Port (RX/TX), and the antenna.

My goal is to access this part to learn more about this object and how it was programmed. In summary, getting to this point seems simple (in reality, it’s not too complicated), but it does require a bit of time, patience, and tools… and frankly, if you modify the object to include a backdoor in this way… I think it will be noticeable.

Tracing boot content

The boot process of the ESP32 occurs in several phases. First, there is a level 1 boot, which is stored in ROM, meaning it is hardcoded into all ESP32s. This starts a level 2 bootloader, which depends on the developer’s code and the SDK they choose to develop with. This level 2 bootloader handles functions such as:

• Loading of firmware from a serial/USB port, for example, for initial injection and recovery.
• Loading data from flash to memory, such as for pre-initialized memory data.
• Encryption of certain memory regions.
• Booting the correct firmware, especially when there are multiple banks to allow for OTA (Over The Air) updates and a potential rollback.

In the case of this device, these boot processes are somewhat verbose and provide more insight into the internal organization of the memory.

First, there is the ROM boot, level 1, which can be found on all ESP32 devices. This provides information, for example, about the reason for the boot, such as the application of a reset signal. After that, we can observe the loading of the bootloader.

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 188777542, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff25a0,len:12988
ho 0 tail 12 room 4
load:0x40078000,len:22336
load:0x40080400,len:13388
entry 0x400807f4

Next, we have the loading of the level 2 bootloader, which is quite verbose here, which isn’t always the case. For example, we can see configuration information, such as the chip running at 80MHz.

I (32) boot: ESP-IDF  2nd stage bootloader
I (32) boot: compile time 00:43:58
I (32) boot: chip revision: 3
I (34) boot_comm: chip revision: 3, min. bootloader chip revision: 0
I (41) qio_mode: Enabling default flash chip QIO
I (47) boot.esp32: SPI Speed      : 80MHz
...

Boot analysis – Flash organization

By conducting a more detailed analysis of the rest of the boot process, we can uncover interesting information about this device and about what has been implemented to protect it or to allow for updates.

Starting with the memory organization, here is a simplified version with fewer details to keep it readable.

I (66) boot: Partition Table:
I (69) boot: ## Label            Usage            ...
I (77) boot:  0 app1_h           unknown          ...
I (84) boot:  1 app1             app              ...
I (92) boot:  2 phy_init         radio            ...
I (99) boot:  3 hsm              unknown          ...
I (107) boot:  4 coredump        Unknown data     ...
I (114) boot:  5 app2_h          unknown          ...
I (122) boot:  6 app2            app              ...
I (129) boot:  7 config          unknown          ...
I (137) boot:  8 wifi            WiFi data        ...
I (144) boot: End of partition table

We can already see that the memory organization has a certain level of complexity. We will find two areas, app1 and app2, which are designed to store the currently used firmware and the future firmware for when the update occurs. There are different approaches to using dual banks; sometimes, the other bank will retain the previous firmware until it’s replaced, and other times, the current firmware is copied to create a backup, preventing an old, potentially vulnerable code from remaining in memory. This logic is tied to the level 2 bootloader, as it is the one that decides which partition to boot from.

Here, we can distinguish some interesting partitions on the device, one of which is called HSM. This likely refers to a hardware security module and could securely store identification elements, such as certificates, that the device will use to connect to web services. These elements are unique to the device and not integrated into the firmware, so they are stored in a memory area that will never be updated.

The coredump is used to store traces of system crashes for later analysis. WiFi is used to store information related to the network configuration required for connection. phy_init is a low-level radio configuration layer. The config area is likely reserved for user settings, ensuring that these configurations remain accessible even after the device is updated.

In the end, with this information, we can reconstruct the organization of the flash memory on the microcontroller across the 4MB available. The blue areas are directly related to the firmware and its two banks. The green areas represent configuration elements tied to the functional aspects of the device, and the yellow areas contain more technical configuration elements, standardized by the ESP SDK and the bootloader used.

Bootloader analysis – device protection

The bootloader contains several other interesting pieces of information. For example, regarding data encryption, the ESP bootloader supports encrypting data in flash, which protects against attacks aimed at extracting data from the flash memory, similar to how a hard drive is encrypted on a PC. Here, we can see that this mechanism must have been enabled, and the areas that need to be encrypted have indeed been encrypted.

I (605) boot: Checking flash encryption...
I (605) flash_encrypt: flash encryption is enabled (0 plaintext flashes left)

In the case of this product, at least the App1, App2, and HSM partitions are encrypted. It is also possible that they are signed. This can be seen in the partition table when we have access to this file. For example, in the partition file shown below, in the framed area, all the values are set to 0x00, which means the partitions are in clear text. The encrypted partitions are marked with 0x01 in this column.

The encryption relies on the eFuses, which allow for storing a key that will no longer be accessible afterward. This key is used to encrypt data when it is written to Flash and enables on-the-fly decryption, performed hardware-side during the data read process. Since the program is executed directly from Flash, it must be decrypted during execution. The encryption is of the AES-256 type, with a salt being added based on the base address of the partition.

To prevent loading custom code onto the ESP32, a secure boot mechanism is enabled, which adds a signature verification step before starting any partition. This signature relies on an asymmetric key mechanism. The person building the partition, whether it’s the bootloader or the application, will use a private key to calculate the signature. All ESP32 devices will have the corresponding public key and will verify the signature. Although this public key is not sensitive by itself, it is stored in the eFuse area and can only be read by the hardware. In the ESP32, an 256bits ECDSA key is used for this purpose with a SHA256 algorithm. It gives a 68 bytes signature, 4 bytes version and 64 bytes digest added to the end of the partition file.

The eFuse memory is 1024bits, 256bits can be used by application.

With the information available so far, it is not possible to definitively determine the exact protection mechanisms in place. However, it is reasonable to assume that signature mechanisms are likely in place; we can imagine that secure boot is also enabled, given that the other mechanisms have been deployed. Having a signature requires a fairly complex build chain to generate the firmware with a private key, which is often stored in an HSM to ensure its security. The bootloader is also encrypted with a slightly different mechanism to ensure it cannot be tampered with either.

One way to check if the signature is in place would be to program an unsigned firmware via the serial port and see if it executes. This approach would be destructive to the device in case signature verification is enabled. However, this is not possible here because another mechanism has been implemented: the serial download has been disabled on the ESP. While I no longer have the screenshot on hand, a clear message appears indicating the deactivation when entering programming mode on the bootloader (by performing a reset with the IO0 pin set to LOW).

Debug logs

In the booloader logs, we can find information about the build dates and versions, as well as the IDs, which are actually the MAC addresses for Wi-Fi and BLE on the device. (The data here has been anonymized.)

I (816) cpu_start: Project name:     lightxxxxx
I (823) cpu_start: App version:      
I (828) cpu_start: Compile time:     Aug 29 2023 16:39:18
I (834) cpu_start: ELF file SHA256:  95911a7f8cf...

Copyright 2016-2022 xxxxx
LRR: 1,0,,,,,,,,,,,,0,0,0,11,254,1,29,1,0,0
IDs: xxxxxxxxx,xxxxxxx,5a
2.5.25478,2.5.25245,2.5.25173 
ASR protocol version: 2

It would generally be preferable to obfuscate this information and, in general, avoid logging of the [I] type in production. However, it’s not a major risk, especially since the device would have had to be destroyed to retrieve this information.

BLE & Wi-Fi sniffing

Now that I’ve more or less completed the hardware analysis of everything directly accessible, I will focus on the data exchanges. The goal is to understand the architecture in place and the interactions between the mobile application, the device, and the servers.

To record and decode BLE traffic, I had a Nordic dongle based on the nRF5142 lying around in one of my cabinets, an oldie but a goodie – perfect!

This small dongle allows capturing BLE traffic, and it works on macOS, Windows, and Linux. A small script (nRF Sniffer) will link it with Wireshark to capture and interpret the packets. Of course, you’ll need to have Wireshark installed to use it.

To install nRF Sniffer, you need to unzip the archive, install the dependencies, and then deploy everything into a directory where Wireshark can find it.

Downloads $ mkdir tmp ; cd tmp
tmp $ unzip ../nrf_sniffer*.zip
tmp $ cd extcap
extcap$ pip install -r requirements.txt
...
extcap$ mkdir -p ~/.local/lib/wireshark/extcap
extcap$ cp -R * ~/.local/lib/wireshark/extcap

$ ./nrf_sniffer_ble.sh

On the nRF dongle, the correct firmware needs to be programmed for capturing traffic. The process is very simple: just drag and drop the firmware onto the drive that appears when you plug in the dongle. You need to select the appropriate firmware for the dongle from the “hex” directory in the archive. For me, it’s sniffer_nrf51dongle_nrf51422.hex.

Now, Wireshark needs to link with these scripts. To do this, go to Wireshark > About Wireshark, select Folders, and make sure the path is correct — it should be the directory where you copied the files (e.g., .local/lib/...). Eventually go into Capture menu and click on Refresh interfaces. You may see in the interface list the nRF sniffer. if you double click on it you mays start seeing the BLE traffic

If you get lost in these tasks, a step by step guide exists following the link.

The capture process is a bit different from a standard network interface: by default, the dongle listens to all devices, and you’ll mainly see broadcasts. When you want to listen to the traffic of a specific device, you need to select it; otherwise, the capture won’t show the data exchange traces. To do this, you need to enable the BLE sniffing toolbar, which is located in View > Interface Toolbar > nRF Sniffer for BLE. The list of devices will grow as new ones are discovered. One of the main challenges with BLE sniffing is often identifying your device among the hundreds of devices you might pick up around you.

Indeed, as we saw earlier in the log, there is the IDs field, which contains both the WiFi and BLE MAC addresses. This can be quite helpful in identifying the correct device when performing the sniffing.

You will indeed need to use the object’s ID by selecting it in order to capture all the traffic related to that specific device. This is essential to ensure you’re tracking the right data exchanges.

In the course of my analysis, I didn’t perform the captures sequentially, but it is interesting to capture all the BLE and WiFi exchanges so they can be associated later. So, even though this article is sequential, it’s better to have everything set up before configuring the device. This way, you’ll have a complete set of data to work with.

Decrypt BLE communication with crackle

During BLE exchanges, there are encryption mechanisms implemented by the protocol that can prevent understanding of the traffic. However, these protocols are partially vulnerable, and it is possible to decrypt them. In the case of analyzing this particular device, the traffic didn’t appear to be encrypted in the BLE sense. However, the data exchanges I analyzed were not interpretable and seemed to be encrypted at the application level. This suggests that, while the BLE layer might not have encryption enabled, the application layer itself is using some form of encryption to secure the communication.

Crackle is a tool that allows you to find the keys used during BLE exchanges and decrypt the communications. It is particularly useful for analyzing encrypted BLE traffic by extracting the encryption keys and then using them to decrypt the data being transmitted.

$ apt install git libpcac-dev
$ git clone https://github.com/mikeryan/crackle.git
$ cd crackle
$ make
$ crackle -i input.pcap/pcapng -o output.pcap

This approach, therefore, yielded no results during my attempts on this device. The application-level encryption likely prevents any meaningful interpretation of the captured data.

Create a WiFi Ap on Linux to capture traffic

As I mentioned earlier, to understand the system and its operation, it is essential to capture all the exchanges and link them together. Therefore, you need to record both the BLE communication between the device and the mobile used for pairing, as well as the WiFi network communication between the mobile and the servers, and between the device and the servers. By doing this, you can correlate the different types of traffic and gain a comprehensive understanding of how the system functions.

I’m using a Raspberry Pi 2W integrated into a breadboard, and I connect to my network via an Ethernet-to-USB port to allow the WiFi controller to function as an access point. I encountered many issues when trying to use an external USB WiFi dongle, so I would recommend the external Ethernet approach instead.

I will configure this Linux machine to act as a WiFi access point named “monitor” with the password “monitor1234”. I will be able to connect both the phone and the device to it in order to capture their network traffic using tcpdump. Then, this capture can be analyzed with Wireshark. While this won’t be sufficient on its own, we will cover this in the next chapter.

$ apt-get install hostapd bridge-utils dnsmasq udhcpd tcpdump
$ killall -KILL dnsmasq
$ nmcli device wifi hotspot ssid monitor password monitor1234 ifname wlan0
$ iptables -A FORWARD -i eth1 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ iptables -A FORWARD -i wlan0 -o eth1 -j ACCEPT
$ iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

Now, it’s possible to record the network traffic on the hotspot and later analyze it using Wireshark. By running tcpdump on the Raspberry Pi

$ tcpdump -i wlan0 -w /home/pi/wlan0.pcap

The issue is that, quite logically, these exchanges will be encrypted, and you won’t learn much more than the names of the servers involved in the communication. To gain deeper insights, it will be necessary to position yourself in the middle of the communication with a transparent proxy (man-in-the-middle). This way, you can decrypt the traffic and analyze the data exchanged between the devices more thoroughly, allowing for a better understanding of the communication flow.

Create a transparent proxy for better traffic sniffing

There is a project perfectly suited for this, called mitmproxy, and its installation is straightforward. For this to work, the device being monitored must allow communication with this proxy, which cannot present a valid certificate for the target domain. To make it work, you’ll need to deploy root certificates on the mobile device and possibly on the object itself to validate the proxy’s certificate as legitimate.

If the device is poorly designed, it may not verify the certificate, and the traffic will be visible. However, in our case, the object is well-designed and does perform this verification. This is another example of complex code management in a device for secure design.

The mitmproxy installation is done the following way:

$ apt install python3-pip python3-pipx iptables
$ pipx install mitmproxy

Now, you can start the service and reroute the traffic to the proxy. This setup is not persistent, meaning that after the next reboot, the Raspberry Pi will return to its normal functionality without the proxy setup. This is useful for temporary traffic monitoring without permanently altering the system’s configuration. You would need to reconfigure the proxy after each reboot unless you create a script or service to automate this process on startup.

$ cd
$ .local/bin/mitmweb --mode transparent --web-port 9090 --web-host 0.0.0.0

$ iptables -t nat -A PREROUTING -i wlan0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080
$ iptables -t nat -A PREROUTING -i wlan0 -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8080

It is now possible to add the mitmproxy root certificate to your phone so that it is no longer rejected by applications. To do this, you need to visit http://mitm.it and download the appropriate certificate for your device. Once downloaded, you must activate it by following the procedure outlined in the associated instructions.

Important: Be sure to disable this function after use, as leaving it enabled could compromise your security by allowing man-in-the-middle attacks on your device. Always remember to remove or revoke the certificate once you’ve finished your analysis to maintain the security of your device.

Once the proxy service is started, the web interface on port 9090 will allow you to track the exchanges in real-time.

$ .local/bin/mitmweb --mode transparent --web-port 9090 --web-host 0.0.0.0 --ignore-hosts gateway.icloud.com 

The traffic will be redirected, and if the application does not accept the certificate, it will not be possible to capture the traffic. However, if the certificate is not verified or is manually accepted. Here is an example of browsing a website after manually accepting the certificate.

This process allows you to intercept and analyze the traffic, but it heavily depends on whether the device or application properly handles certificate validation. If it does not, you can gain access to the decrypted traffic. Even with the root certificate injected into the mobile, some communications are rejected, possibly because the application does not use the phone’s CA. This is why there is an --ignore-hosts entry in the command line; it helps avoid too many errors.

As part of the device analysis, I was able to gather a lot of information about the cloud architecture and the roles of the various servers. I also noticed that for certain functions, the mobile application included its own CA, possibly for a higher level of protection. In any case, some functions were no longer accessible without bypassing certain cloud services, while other parts of the application were still communicating with the same service openly under my observation.

This suggests that the application has some form of segmentation in its communication paths, possibly as a security measure to prevent unauthorized access to sensitive services or to ensure that only trusted communication channels are used for specific actions. It’s a good example of how complex security mechanisms can be embedded within an application to protect certain features from exposure.

Add Mqtt support for mitmproxy

The following project allows to analyse MQTTs as MITM with mitmproxy https://github.com/nikitastupin/mitmproxy-mqtt-script

Rewrite an url dynamically

As part of my analysis, as I mentioned, the device rejected the connections to the proxy due to the invalid certificate presented, so I couldn’t intercept its traffic. However, by presenting an invalid certificate, the device assumed that perhaps its own root certificates were incorrect. It then proceeded to download an update of its root CAs from the solution’s cloud servers. It was therefore interesting to see if it would be possible to inject my own certificate through this process.

mitmproxy allows you to write scripts that modify the exchanges. For example, it is possible to capture a request to a server and redirect it to another server, thus forcing my device to download my own CA certificate bundle.

A script exists on mitmproxy on example/add-on directory on github. You can complete the redirection rules you want into it and run mitmproxy with this filter (redirect.py).

from mitmproxy import http


def request(flow: http.HTTPFlow) -> None:
    # pretty_host takes the "Host" header of the request into account,
    # which is useful in transparent mode where we usually only have the IP
    # otherwise.
    if flow.request.pretty_host == "www.foo.bar":
        flow.request.host = "www.disk91.com"

Run mitm proxy with the redirect filter

.local/bin/mitmweb --mode transparent --web-port 9090 --web-host 0.0.0.0 -s redirect.py

Here is another security mechanism to implement in a connected device. My attempt to make it load my file worked, but the file itself was not accepted by the device. Upon analyzing the CA bundle used by the device, it seems that the header contains signatures that authenticate the creator of the bundle. Not only do I have no information about the format of this header, which doesn’t follow any of the documentation found on ESP, but I also have no access to the private key that was likely used to sign the file.

A point of attention: everything I found online was not protected against such an attack, but the device’s manufacturer has indeed taken this risk into account and implemented proper security measures.

Modify the Certificate bundle

To create my own certificate bundle, I had to embed a certificate in text format within a binary file. Along the way, I discovered a great tool on Mac for manipulating hexadecimal files, somewhat like working with text files : HexFriend

Overall analysis

With these different methods, I gained a solid overall view of the ecosystem surrounding the device, and the network analysis provided interesting insights that highlight that IoT is primarily about sophisticated cloud platforms. From a firmware perspective, what I saw is that numerous security mechanisms are implemented, even on a device like a light bulb, where the business logic may only consist of a few hundred lines of code. In the end, there will be tens of thousands of lines written to handle updates, secure communication with the cloud, and more.

I still have some work to do on this device to fully understand it, with a few leads to uncover more of its hidden secrets. However, at this point, we are starting to move beyond what I can publish about this experience, as the next steps will involve less of a “black box” approach.

Thanks to these approaches that anyone can undertake, here’s a summary of what I can determine about the cloud architecture and the exchanges:

In short, upon startup, a device will check if it has network connectivity by contacting the connectivity test server. It will then synchronize its clock with time servers, which I imagine is primarily to ensure the validity of the certificates—yet another complexity that one might not necessarily consider.

If that’s the case, the device will be able to connect to the broker infrastructure, which is a cluster and operates with HTTP2 streaming. From there, it can receive commands sent from mobile devices in almost real-time. If it cannot connect or if the date indicates that its certificates have expired, it will download a new certificate bundle from an unencrypted content server. However, as we saw, these certificates, which are only public keys, are signed to ensure their integrity.

I’ve identified two broker-type infrastructures in use, either one or the other. The first seems to consist of a single member, while the other is a cluster. This could potentially be an adaptation to the current load of the service.

I haven’t identified the trigger for accessing the OTA service yet. It’s possible that it occurs after a version exchange with the broker, whose role is likely more important than just being an integration component, and it might actually handle business logic, such as for updates. Indeed, when studying different devices, I noticed varying behaviors regarding updates. For now, I’ve preferred to block the update on the device that triggers it so that I can reproduce these exchanges.The application interacts both with the device via BLE for the initial configuration at least, and with the device over the network through the streamed flow via the broker for direct communication with the device. This allows user requests to be processed within a few hundred milliseconds, regardless of the distance from the device.

With each update, the device’s reference is updated. This service also seems to enable the description of the application’s interface and the active functionalities within the device. It is, in a way, a digital twin of the device—an interesting approach that likely allows for immediate reconciliation of the current state of the device, even as multiple users can interact with it across different applications.