BLESession.setNotifications: Lookup CCCD to start or stop notifications

Per Bluetooth Core Specification Vol.3, Part G, Section 3.3.1.1, if
Notify property is set for a Characteristic, the Client Characteristic
Configuration Descriptor, or CCCD, shall exist. As noted in section
3.3.3.3 of the specification, notification of the Characteristic is
enabled or disabled by setting bit 0 of the CCCD.

This CCCD exists within the characteristic definition following the
Characteristic Value declaration, and section 3.3.3 of the specification
warns that the order of characteristic declaration shall not be assumed.
However, current pyscrlink code assumes that the CCCD comes just after
the Characteristic Value and CCCD handle is obtained just adding 1 to
the handle of the Characteristic Value. This works many of the BLE
devices but does not work all of them.

To get correct CCCD handle, look up descriptors of the Characteristic to
find out the CCCD.

Signed-off-by: Shin'ichiro Kawasaki <kawasaki@juno.dti.ne.jp>

.gitignore

@@ -1,5 +1 @@
-__pycache__
+/__pycache__
-build
-dist
-pyscrlink/__pycache__
-pyscrlink.egg-info

README.md

@@ -9,16 +9,11 @@ micro:bit.
 
 Pyscrlink allows you to connect Scratch and bluetooth devices with the Linux
 OSes. It uses the Linux Bluetooth protocol stack [Bluez](http://www.bluez.org/)
-and [bluepy](https://github.com/IanHarvey/bluepy) to handle Bluetooth Low Energy
+and its python interfaces [pybluez](https://github.com/pybluez/pybluez) to
-(BLE) connections. It has been reported that pyscrlink connects Scratch 3.0 with
+handle Bluetooth, and [bluepy](https://github.com/IanHarvey/bluepy) to handle
-micro:bit, LEGO WeDo, LEGO Boost and toio.
+Bluetooth Low Energy (BLE) connections. It has been reported that pyscrlink
-
+connects Scratch 3.0 with micro:bit, LEGO Mindstorms EV3, LEGO WeDo, LEGO
-Until version v0.2.5, pyscrlink supported Bluetooth Classic protocol using
+Boost, Intelino Smart Train and toio.
-[pybluez](https://github.com/pybluez/pybluez). Unfortunately, pybluez is not
-well maintained and caused technical troubles. Then Bluetooth Classic protocol
-support is dropped from pyscrlink. This means that LEGO Mindstorm EV3 can not
-be connected with pyscrlink. Bluetooth Classic support is the improvement
-opportunity of pyscrlink.
 
 To use websockets, pyscrlink requires python version 3.6 or later. If your
 system has python older than version 3.6, install newer version. If your
@@ -44,6 +39,7 @@ Devices:
 
 Linux distros:
 * Arch Linux
+* elementary OS 5.1 Hera
 
 Browsers:
 * Firefox
@@ -53,8 +49,10 @@ It was reported that pyscrlink (former bluepy-scratch-link) working with
 following devices and Linux distros.
 
 Devices:
+* LEGO Mindstorm EV3 by @chrisglencross
 * LEGO WeDo by @zhaowe, @KingBBQ
 * LEGO Boost and compatible devices by @laurentchar, @miguev, @jacquesdt, @n3storm
+* Intelino Smart Train by @ErrorJan
 * toio by @shimodash
 
 Linux distros:
@@ -91,6 +89,7 @@ Installation
     ```
 
 4. Set bluepy-helper capability.
+   This step is required for most of devices except LEGO Mindstorm EV3.
 
     ```
     $ bluepy_helper_cap
@@ -109,18 +108,67 @@ Installation
 
 Usage
 -----
-1. Start scratch-link python script.
+1. For LEGO Mindstorms EV3, pair your Linux PC to the EV3 brick.
+
+   First, turn on the EV3 and ensure Bluetooth is enabled.
+
+   Then, pair using your Linux desktop's the Bluetooth settings.
+
+   If using Gnome:
+      * Settings -> Bluetooth
+      * Click on the EV3 device name
+      * Accept the connection on EV3 brick
+      * Enter a matching PIN on EV3 brick and Linux PC. '1234' is the value Scratch suggests.
+      * Confirm EV3 status is "Disconnected" in Bluetooth settings
+
+   With a Raspberry Pi default Raspbian desktop, click the Bluetooth logo in the top right of the screen and
+   Add Device. Then follow the Gnome instructions. You will be warned that the Raspberry Pi
+   does not know how to talk to this device; that is not a problem.
+
+   Alternatively you can perform pairing from the command-line:
+   ```shell script
+   $ bluetoothctl
+
+   [bluetooth]# power on
+   Changing power on succeeded
+
+   [bluetooth]# pairable on
+   Changing pairable on succeeded
+
+   [bluetooth]# agent KeyboardOnly
+   Agent registered
+
+   [bluetooth]# devices
+   ...
+   Device 00:16:53:53:D3:19 EV3
+   ...
+
+   [bluetooth]# pair 00:16:53:53:D3:19
+   Attempting to pair with 00:16:53:53:D3:19
+
+   # Confirm pairing on the EV3 display, set PIN to 1234
+
+   Request PIN code
+   [agent] Enter PIN code: 1234
+   [CHG] Device 00:16:53:53:D3:19 Connected: yes
+   [CHG] Device 00:16:53:53:D3:19 Paired: yes
+   Pairing successful
+
+   [bluetooth]# quit
+   ```
+
+2. Start scratch-link python script.
     ```sh
     $ scratch_link
     ```
     If your device is toio, add "-s 1" option to the scratch_link command. It
     allows the toio Do Visual Programming to connect to toio automatically.
 
-2. Connect scratch to the target device such as micro:bit:
+3. Connect scratch to the target device such as micro:bit or LEGO Mindstorms:
     * Open FireFox or Chrome. (Make sure to run as the same user for scratch-link python script.)
     * Access [Scratch 3.0](https://scratch.mit.edu/) and create your project.
     * Select the "Add Extension" button.
-    * Select the extension for your device (e.g., micro:bit) and follow the prompts to connect.
+    * Select the extension for your device (e.g., micro:bit or Lego Mindstorms EV3 extension) and follow the prompts to connect.
     * Build your project with the extension blocks.
 
 In Case You Fail to Connect
@@ -158,14 +206,6 @@ Please file issues to [GitHub issue tracker](https://github.com/kawasaki/pyscrli
 Releases
 --------
 
-Release 0.2.6
-
-* Removed Bluetooth Classic and LEGO Mindstorm EV3 support
-
-Release 0.2.5
-
-* Fixed handling of multiple UUIDs for LEGO Boost
-
 Release 0.2.4
 
 * Added -s option to specify BLE scan duration

pyscrlink/scratch_link.py

@@ -17,6 +17,9 @@ import signal
 import traceback
 import argparse
 
+# for Bluetooth (e.g. Lego EV3)
+import bluetooth
+
 # for BLESession (e.g. BBC micro:bit)
 from bluepy.btle import Scanner, UUID, Peripheral, DefaultDelegate
 from bluepy.btle import BTLEDisconnectError, BTLEManagementError
@@ -44,6 +47,8 @@ logger.propagate = False
 HOSTNAME="device-manager.scratch.mit.edu"
 scan_seconds=10.0
 
+CCCD_UUID = 0x2902
+
 class Session():
     """Base class for BTSession and BLESession"""
     def __init__(self, websocket, loop):
@@ -135,6 +140,204 @@ class Session():
 class BTSession(Session):
     """Manage a session for Bluetooth device"""
 
+    INITIAL = 1
+    DISCOVERY = 2
+    DISCOVERY_COMPLETE = 3
+    CONNECTED = 4
+    DONE = 5
+
+    # Split this into discovery thread and communication thread
+    # discovery thread should auto-terminate
+
+    class BTThread(threading.Thread):
+        """
+        Separated thread to control notifications to Scratch.
+        It handles device discovery notification in DISCOVERY status
+        and notifications from bluetooth devices in CONNECTED status.
+        """
+
+        class BTDiscoverer(bluetooth.DeviceDiscoverer):
+
+            def __init__(self, major_class, minor_class):
+                super().__init__()
+                self.major_class = major_class
+                self.minor_class = minor_class
+                self.found_devices = {}
+                self.done = False
+
+            def pre_inquiry(self):
+                self.done = False
+
+            def device_discovered(self, address, device_class, rssi, name):
+                logger.debug(f"Found device {name} addr={address} class={device_class} rssi={rssi}")
+                major_class = (device_class & 0x1F00) >> 8
+                minor_class = (device_class & 0xFF) >> 2
+                if "LEGO Hub" in name:
+                    minor_class = 1
+                    logger.info(f"Pretend to be LEGO EV3 with LEGO Hub: class={major_class/minor_class}")
+                if major_class == self.major_class and minor_class == self.minor_class:
+                    self.found_devices[address] = (name, device_class, rssi)
+
+            def inquiry_complete(self):
+                self.done = True
+
+        def __init__(self, session, major_device_class, minor_device_class):
+            threading.Thread.__init__(self)
+            self.session = session
+            self.major_device_class = major_device_class
+            self.minor_device_class = minor_device_class
+            self.cancel_discovery = False
+            self.ping_time = None
+
+        def discover(self):
+            major = self.major_device_class
+            minor = self.minor_device_class
+            if major == 8 and minor == 1:
+                logger.info(f"Search LEGO Hub instead of LEGO EV3")
+                minor = 4
+            logger.debug(f"BT discover: class={major}/{minor}")
+            discoverer = self.BTDiscoverer(major, minor)
+            discoverer.find_devices(lookup_names=True)
+            while self.session.status == self.session.DISCOVERY and not discoverer.done and not self.cancel_discovery:
+                readable = select.select([discoverer], [], [], 0.5)[0]
+                if discoverer in readable:
+                    discoverer.process_event()
+                    for addr, (device_name, device_class, rssi) in discoverer.found_devices.items():
+                        logger.debug(f"notifying discovered {addr}: {device_name}")
+                        params = {"rssi": rssi, 'peripheralId': addr, 'name': device_name.decode("utf-8")}
+                        self.session.notify('didDiscoverPeripheral', params)
+                    discoverer.found_devices.clear()
+
+            if not discoverer.done:
+                discoverer.cancel_inquiry()
+
+        def run(self):
+            while self.session.status != self.session.DONE:
+
+                logger.debug(f"loop in BT thread: session status={self.session.status}")
+                current_time = int(round(time.time()))
+
+                if self.session.status == self.session.DISCOVERY and not self.cancel_discovery:
+                    logger.debug("in discovery status:")
+                    try:
+                        self.discover()
+                        self.ping_time = current_time + 5
+                    finally:
+                        self.session.status = self.session.DISCOVERY_COMPLETE
+
+                elif self.session.status == self.session.CONNECTED:
+                    logger.debug("in connected status:")
+                    sock = self.session.sock
+                    try:
+                        ready = select.select([sock], [], [], 1)
+                        if ready[0]:
+                            header = sock.recv(2)
+                            [msg_len] = struct.unpack("<H", header)
+                            msg_data = sock.recv(msg_len)
+                            data = header + msg_data
+                            params = {'message': base64.standard_b64encode(data).decode('utf-8'), "encoding": "base64"}
+                            self.session.notify('didReceiveMessage', params)
+                            self.ping_time = current_time + 5
+
+                    except Exception as e:
+                            logger.error(e)
+                            self.session.close()
+                            break
+
+                    # To avoid repeated lock by this single thread,
+                    # yield CPU to other lock waiting threads.
+                    time.sleep(0)
+                else:
+                    # Nothing to do:
+                    time.sleep(1)
+
+                # Terminate if we have lost websocket connection to Scratch (e.g. browser closed)
+                if self.ping_time is None or self.ping_time <= current_time:
+                    try:
+                        self.session.notify('ping', {})
+                        self.ping_time = current_time + 5
+                    except Exception as e:
+                        logger.error(e)
+                        self.session.close()
+                        break
+
+    def __init__(self, websocket, loop):
+        super().__init__(websocket, loop)
+        self.status = self.INITIAL
+        self.sock = None
+        self.bt_thread = None
+
+    def close(self):
+        self.status = self.DONE
+        if self.sock:
+            logger.info(f"disconnect to BT socket: {self.sock}")
+            self.sock.close()
+            self.sock = None
+
+    def __del__(self):
+        self.close()
+
+    def handle_request(self, method, params):
+        """Handle requests from Scratch"""
+        logger.debug("handle request to BT device")
+        logger.debug(method)
+        if len(params) > 0:
+            logger.debug(params)
+
+        res = { "jsonrpc": "2.0" }
+
+        if self.status == self.INITIAL and method == 'discover':
+            logger.debug("Starting async discovery")
+            self.status = self.DISCOVERY
+            self.bt_thread = self.BTThread(self, params["majorDeviceClass"], params["minorDeviceClass"])
+            self.bt_thread.start()
+            res["result"] = None
+
+        elif self.status in [self.DISCOVERY, self.DISCOVERY_COMPLETE] and method == 'connect':
+
+            # Cancel discovery
+            while self.status == self.DISCOVERY:
+                logger.debug("Cancelling discovery")
+                self.bt_thread.cancel_discovery = True
+                time.sleep(1)
+
+            addr = params['peripheralId']
+            logger.debug(f"connecting to the BT device {addr}")
+            try:
+                self.sock = bluetooth.BluetoothSocket(bluetooth.RFCOMM)
+                self.sock.connect((addr, 1))
+                logger.info(f"connected to BT device: {addr}")
+            except bluetooth.BluetoothError as e:
+                logger.error(f"failed to connect to BT device: {e}", exc_info=e)
+                self.status = self.DONE
+                self.sock = None
+
+            if self.sock:
+                res["result"] = None
+                self.status = self.CONNECTED
+            else:
+                err_msg = f"BT connect failed: {addr}"
+                res["error"] = { "message": err_msg }
+                self.status = self.DONE
+
+        elif self.status == self.CONNECTED and method == 'send':
+            logger.debug("handle send request")
+            if params['encoding'] != 'base64':
+                logger.error("encoding other than base 64 is not "
+                                 "yet supported: ", params['encoding'])
+            msg_bstr = params['message'].encode('ascii')
+            data = base64.standard_b64decode(msg_bstr)
+            self.sock.send(data)
+            res['result'] = len(data)
+
+        logger.debug(res)
+        return res
+
+    def end_request(self):
+        logger.debug(f"end_request of BTSession {self}")
+        return self.status == self.DONE
+
+
 class BLESession(Session):
     """
     Manage a session for Bluetooth Low Energy device such as micro:bit
@@ -161,6 +364,14 @@ class BLESession(Session):
     scan_lock = threading.RLock()
     scan_started = False
 
+    @staticmethod
+    def _getDevName(dev):
+        """
+        Get AdType 0x09 (Completed local name). If it is not available,
+        get AdType 0x08 (Shortened local name).
+        """
+        return dev.getValueText(0x9) or dev.getValueText(0x8)
+
     class BLEThread(threading.Thread):
         """
         Separated thread to control notifications to Scratch.
@@ -180,7 +391,7 @@ class BLESession(Session):
                     for d in devices:
                         params = { 'rssi': d.rssi }
                         params['peripheralId'] = devices.index(d)
-                        params['name'] = d.getValueText(0x9) or d.getValueText(0x8)
+                        params['name'] = BLESession._getDevName(d)
                         self.session.notify('didDiscoverPeripheral', params)
                     time.sleep(1)
                 elif self.session.status == self.session.CONNECTED:
@@ -197,7 +408,8 @@ class BLESession(Session):
                                 logger.debug("after waitForNotification")
                             logger.debug("released lock for waitForNotification")
                         except Exception as e:
-                            logger.error(e)
+                            logger.error(f"Exception in waitForNotifications: "
+                                         f"{type(e).__name__}: {e}")
                             self.session.close()
                             break
                     else:
@@ -221,7 +433,8 @@ class BLESession(Session):
             self.restart_notification_event.set()
 
         def add_handle(self, serviceId, charId, handle):
-            logger.debug(f"add handle for notification: {handle}")
+            logger.debug(f"add handle for notification: "
+                         f"{serviceId} {charId} {handle}")
             params = { 'serviceId': UUID(serviceId).getCommonName(),
                        'characteristicId': charId,
                        'encoding': 'base64' }
@@ -229,6 +442,14 @@ class BLESession(Session):
 
         def handleNotification(self, handle, data):
             logger.debug(f"BLE notification: {handle} {data}")
+            if handle not in self.handles:
+                logger.error(f"Notification with unknown handle: {handle}")
+                keys = list(self.handles.keys())
+                if keys and len(keys) == 1:
+                    logger.debug(f"Debug: override {handle} with {keys[0]}")
+                    handle = keys[0]
+                else:
+                    return
             params = self.handles[handle].copy()
             params['message'] = base64.standard_b64encode(data).decode('ascii')
             self.session.notify('characteristicDidChange', params)
@@ -293,7 +514,7 @@ class BLESession(Session):
                             return True
             if 'namePrefix' in f:
                 # 0x08: Shortened Local Name
-                deviceName = dev.getValueText(0x08)
+                deviceName = self._getDevName(dev)
                 if not deviceName:
                     continue
                 logger.debug(f"Name of \"{deviceName}\" begins with: \"{f['namePrefix']}\"?")
@@ -409,7 +630,7 @@ class BLESession(Session):
         elif self.status == self.DISCOVERY and method == 'connect':
             logger.debug("connecting to the BLE device")
             self.device = BLESession.found_devices[params['peripheralId']]
-            self.deviceName = self.device.getValueText(0x9) or self.device.getValueText(0x8)
+            self.deviceName = self._getDevName(self.device)
             try:
                 self.perip = Peripheral(self.device)
                 logger.info(f"connected to the BLE peripheral: {self.deviceName}")
@@ -486,11 +707,19 @@ class BLESession(Session):
         service = self._get_service(service_id)
         c = self._get_characteristic(chara_id)
         handle = c.getHandle()
+        # get CCCD or Client Characterstic Configuration Descriptor
+        cccd = None
+        for d in c.getDescriptors():
+            if d.uuid == UUID(CCCD_UUID):
+                cccd = d
+        if not cccd:
+            logger.error("Characteristic {char_id} does not have CCCD")
+            return
         # prepare notification handler
         self.delegate.add_handle(service_id, chara_id, handle)
         # request notification to the BLE device
         with self.lock:
-            self.perip.writeCharacteristic(handle + 1, value, True)
+            self.perip.writeCharacteristic(cccd.handle, value, True)
 
     def startNotifications(self, service_id, chara_id):
         logger.debug(f"start notification for {chara_id}")

release.sh

@@ -10,7 +10,7 @@ usage() {
 
 case ${1} in
 	build)
-		python -m build
+		python setup.py sdist bdist_wheel
 		;;
 	upload)
 		shift

setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setuptools.setup(
     name="pyscrlink",
-    version="0.2.6",
+    version="0.2.4",
     author="Shin'ichiro Kawasaki",
     author_email='kawasaki@juno.dti.ne.jp',
     description='Scratch-link for Linux with Python',
@@ -22,6 +22,7 @@ setuptools.setup(
     install_requires=[
         'websockets',
         'bluepy',
+        'pybluez',
         'pyOpenSSL',
         ],
     entry_points={