BLESession.setNotifications: Lookup CCCD to start or stop notifications

Per Bluetooth Core Specification Vol.3, Part G, Section 3.3.1.1, if
Notify property is set for a Characteristic, the Client Characteristic
Configuration Descriptor, or CCCD, shall exist. As noted in section
3.3.3.3 of the specification, notification of the Characteristic is
enabled or disabled by setting bit 0 of the CCCD.

This CCCD exists within the characteristic definition following the
Characteristic Value declaration, and section 3.3.3 of the specification
warns that the order of characteristic declaration shall not be assumed.
However, current pyscrlink code assumes that the CCCD comes just after
the Characteristic Value and CCCD handle is obtained just adding 1 to
the handle of the Characteristic Value. This works many of the BLE
devices but does not work all of them.

To get correct CCCD handle, look up descriptors of the Characteristic to
find out the CCCD.

Signed-off-by: Shin'ichiro Kawasaki <kawasaki@juno.dti.ne.jp>

README.md

@@ -13,7 +13,7 @@ and its python interfaces [pybluez](https://github.com/pybluez/pybluez) to
 handle Bluetooth, and [bluepy](https://github.com/IanHarvey/bluepy) to handle
 Bluetooth Low Energy (BLE) connections. It has been reported that pyscrlink
 connects Scratch 3.0 with micro:bit, LEGO Mindstorms EV3, LEGO WeDo, LEGO
-Boost and Intelino Smart Train.
+Boost, Intelino Smart Train and toio.
 
 To use websockets, pyscrlink requires python version 3.6 or later. If your
 system has python older than version 3.6, install newer version. If your
@@ -42,7 +42,7 @@ Linux distros:
 * elementary OS 5.1 Hera
 
 Browsers:
-* FireFox
+* Firefox
 * Chromium
 
 It was reported that pyscrlink (former bluepy-scratch-link) working with
@@ -51,13 +51,15 @@ following devices and Linux distros.
 Devices:
 * LEGO Mindstorm EV3 by @chrisglencross
 * LEGO WeDo by @zhaowe, @KingBBQ
-* LEGO Boost by @laurentchar, @miguev, @jacquesdt
+* LEGO Boost and compatible devices by @laurentchar, @miguev, @jacquesdt, @n3storm
 * Intelino Smart Train by @ErrorJan
+* toio by @shimodash
 
 Linux distros:
 * Raspbian by @chrisglencross
 * Ubuntu 16.04 @jacquesdt
 * Ubuntu Studio 20.04 @miguev
+* Debian 11 @n3storm
 
 Installation
 ------------
@@ -73,7 +75,7 @@ Installation
 
     ```sh
     Ubuntu
-    $ sudo apt install bluez libbluetooth-dev libnss3-tools libcap2-bin
+    $ sudo apt install bluez libbluetooth-dev libnss3-tools libcap2-bin libglib2.0-dev
     Arch
     $ sudo pacman -S bluez bluez-utils nss libcap
     ```
@@ -86,7 +88,8 @@ Installation
     $ pip3 install pyscrlink
     ```
 
-4. For Bluetooth Low Energy (BLE) devices, set bluepy-helper capability.
+4. Set bluepy-helper capability.
+   This step is required for most of devices except LEGO Mindstorm EV3.
 
     ```
     $ bluepy_helper_cap
@@ -99,7 +102,7 @@ Installation
 5. For micro:bit, install Scratch-link hex on your device.
 
     * Download and unzip the [micro:bit Scratch Hex file](https://downloads.scratch.mit.edu/microbit/scratch-microbit-1.1.0.hex.zip).
-    * Flash the micro:bit over USB with the Scratch .Hex File, you will see the
+    * Flash the micro:bit over USB with the Scratch Hex File, you will see the
       five character name of the micro:bit scroll across the screen such as
       'zo9ev'.
 
@@ -158,8 +161,10 @@ Usage
     ```sh
     $ scratch_link
     ```
+    If your device is toio, add "-s 1" option to the scratch_link command. It
+    allows the toio Do Visual Programming to connect to toio automatically.
 
-3. Connect scratch to micro:bit or LEGO Mindstorms:
+3. Connect scratch to the target device such as micro:bit or LEGO Mindstorms:
     * Open FireFox or Chrome. (Make sure to run as the same user for scratch-link python script.)
     * Access [Scratch 3.0](https://scratch.mit.edu/) and create your project.
     * Select the "Add Extension" button.
@@ -187,6 +192,12 @@ In Case You Fail to Connect
 3. If scratch_link.py says "failed to connect to BT device: [Errno 13] Permission denied",
    make sure to pair the bluetooth device to your PC before connecting to Scratch.
 
+4. To connect to multiple devices at the same time, make all the target devices
+   ready for scan at the first device scan. This is important for toio. The toio
+   allows a single project to connect to two toio devices.
+   * When the second device was prepared after the first device was connected, device scan can not find the second device.
+   * To scan and find the second device, disconnect connections for the first device beforehand.
+
 Issus Reporting
 ---------------
 
@@ -195,6 +206,25 @@ Please file issues to [GitHub issue tracker](https://github.com/kawasaki/pyscrli
 Releases
 --------
 
+Release 0.2.4
+
+* Added -s option to specify BLE scan duration
+* Improved README.md
+
+Release 0.2.3
+
+* Fixed eternal loop caused by hostname resolve failure
+
+Release 0.2.2
+
+* Supported multiple device connections for toio
+* Improved session closure handling
+
+Release 0.2.1
+
+* Added libglib to required package list in README.md
+* Improved setcap and getcap tool finding
+
 Release 0.2.0
 
 * Latency issue fix for BLE devices' write characteristics

pyscrlink/bluepy_helper_cap.py

@@ -22,10 +22,26 @@ logger.propagate = False
 # Check dependent tools
 DEPENDENT_TOOLS = {
     "setcap": "libcap2-bin (Ubuntu) or libcap (Arch)",
+    "getcap": "libcap2-bin (Ubuntu) or libcap (Arch)",
 }
 
+tools = {}
+
 for cmd in DEPENDENT_TOOLS:
-    if not shutil.which(cmd):
+    # find the tools in PATH
+    path = shutil.which(cmd)
+    if path:
+        tools[cmd] = path
+        logger.debug(f"{cmd} found: {path}")
+        continue
+    # find the tools out of PATH but in major directories
+    for d in ["/usr/bin", "/bin", "/usr/sbin", "/sbin"]:
+        path = d + '/' + cmd
+        if os.path.isfile(path) and os.access(path, os.X_OK):
+            tools[cmd] = path
+            logger.debug(f"{cmd} found: {path}")
+            break
+    if not cmd in tools:
         print(f"'{cmd}' not found. Install package {DEPENDENT_TOOLS[cmd]}.")
         sys.exit(1)
 
@@ -42,7 +58,7 @@ def helper_path():
 
 def is_set():
     path = helper_path()
-    p = subprocess.run(["getcap", path], stdout=subprocess.PIPE)
+    p = subprocess.run([tools["getcap"], path], stdout=subprocess.PIPE)
     if p.returncode != 0:
         logger.error(f"Failed to get capability of {path}")
         return False
@@ -53,8 +69,8 @@ def setcap():
     path = helper_path()
     if is_set():
         return True
-    p = subprocess.run(["sudo", "setcap", "cap_net_raw,cap_net_admin+eip",
-                        path])
+    p = subprocess.run(["sudo", tools["setcap"],
+                        "cap_net_raw,cap_net_admin+eip", path])
     if p.returncode !=0:
         logger.error(f"Failed to set capability to {path}")
         return False

pyscrlink/scratch_link.py

@@ -8,12 +8,14 @@ import asyncio
 import pathlib
 import ssl
 import websockets
+import socket
 import json
 import base64
 import logging
 import sys
 import signal
 import traceback
+import argparse
 
 # for Bluetooth (e.g. Lego EV3)
 import bluetooth
@@ -42,6 +44,11 @@ logger.setLevel(logLevel)
 logger.addHandler(handler)
 logger.propagate = False
 
+HOSTNAME="device-manager.scratch.mit.edu"
+scan_seconds=10.0
+
+CCCD_UUID = 0x2902
+
 class Session():
     """Base class for BTSession and BLESession"""
     def __init__(self, websocket, loop):
@@ -165,6 +172,9 @@ class BTSession(Session):
                 logger.debug(f"Found device {name} addr={address} class={device_class} rssi={rssi}")
                 major_class = (device_class & 0x1F00) >> 8
                 minor_class = (device_class & 0xFF) >> 2
+                if "LEGO Hub" in name:
+                    minor_class = 1
+                    logger.info(f"Pretend to be LEGO EV3 with LEGO Hub: class={major_class/minor_class}")
                 if major_class == self.major_class and minor_class == self.minor_class:
                     self.found_devices[address] = (name, device_class, rssi)
 
@@ -180,7 +190,13 @@ class BTSession(Session):
             self.ping_time = None
 
         def discover(self):
-            discoverer = self.BTDiscoverer(self.major_device_class, self.minor_device_class)
+            major = self.major_device_class
+            minor = self.minor_device_class
+            if major == 8 and minor == 1:
+                logger.info(f"Search LEGO Hub instead of LEGO EV3")
+                minor = 4
+            logger.debug(f"BT discover: class={major}/{minor}")
+            discoverer = self.BTDiscoverer(major, minor)
             discoverer.find_devices(lookup_names=True)
             while self.session.status == self.session.DISCOVERY and not discoverer.done and not self.cancel_discovery:
                 readable = select.select([discoverer], [], [], 0.5)[0]
@@ -198,7 +214,7 @@ class BTSession(Session):
         def run(self):
             while self.session.status != self.session.DONE:
 
-                logger.debug("loop in BT thread")
+                logger.debug(f"loop in BT thread: session status={self.session.status}")
                 current_time = int(round(time.time()))
 
                 if self.session.status == self.session.DISCOVERY and not self.cancel_discovery:
@@ -343,6 +359,19 @@ class BLESession(Session):
 
     MAX_SCANNER_IF = 3
 
+    found_devices = []
+    nr_connected = 0
+    scan_lock = threading.RLock()
+    scan_started = False
+
+    @staticmethod
+    def _getDevName(dev):
+        """
+        Get AdType 0x09 (Completed local name). If it is not available,
+        get AdType 0x08 (Shortened local name).
+        """
+        return dev.getValueText(0x9) or dev.getValueText(0x8)
+
     class BLEThread(threading.Thread):
         """
         Separated thread to control notifications to Scratch.
@@ -358,11 +387,11 @@ class BLESession(Session):
                 logger.debug("loop in BLE thread")
                 if self.session.status == self.session.DISCOVERY:
                     logger.debug("send out found devices")
-                    devices = self.session.found_devices
+                    devices = BLESession.found_devices
                     for d in devices:
                         params = { 'rssi': d.rssi }
                         params['peripheralId'] = devices.index(d)
-                        params['name'] = d.getValueText(0x9) or d.getValueText(0x8)
+                        params['name'] = BLESession._getDevName(d)
                         self.session.notify('didDiscoverPeripheral', params)
                     time.sleep(1)
                 elif self.session.status == self.session.CONNECTED:
@@ -379,7 +408,8 @@ class BLESession(Session):
                                 logger.debug("after waitForNotification")
                             logger.debug("released lock for waitForNotification")
                         except Exception as e:
-                            logger.error(e)
+                            logger.error(f"Exception in waitForNotifications: "
+                                         f"{type(e).__name__}: {e}")
                             self.session.close()
                             break
                     else:
@@ -403,7 +433,8 @@ class BLESession(Session):
             self.restart_notification_event.set()
 
         def add_handle(self, serviceId, charId, handle):
-            logger.debug(f"add handle for notification: {handle}")
+            logger.debug(f"add handle for notification: "
+                         f"{serviceId} {charId} {handle}")
             params = { 'serviceId': UUID(serviceId).getCommonName(),
                        'characteristicId': charId,
                        'encoding': 'base64' }
@@ -411,6 +442,14 @@ class BLESession(Session):
 
         def handleNotification(self, handle, data):
             logger.debug(f"BLE notification: {handle} {data}")
+            if handle not in self.handles:
+                logger.error(f"Notification with unknown handle: {handle}")
+                keys = list(self.handles.keys())
+                if keys and len(keys) == 1:
+                    logger.debug(f"Debug: override {handle} with {keys[0]}")
+                    handle = keys[0]
+                else:
+                    return
             params = self.handles[handle].copy()
             params['message'] = base64.standard_b64encode(data).decode('ascii')
             self.session.notify('characteristicDidChange', params)
@@ -418,7 +457,6 @@ class BLESession(Session):
     def __init__(self, websocket, loop):
         super().__init__(websocket, loop)
         self.status = self.INITIAL
-        self.found_devices = []
         self.device = None
         self.deviceName = None
         self.perip = None
@@ -426,6 +464,13 @@ class BLESession(Session):
         self.characteristics_cache = []
 
     def close(self):
+        if self.status == self.CONNECTED:
+            BLESession.nr_connected -= 1
+            logger.info(f"BLE session disconnected")
+            logger.debug(f"BLE session connected={BLESession.nr_connected}")
+            if BLESession.nr_connected == 0:
+                logger.info("all BLE sessions disconnected")
+                BLESession.scan_started = False
         self.status = self.DONE
         if self.perip:
             logger.info("disconnect from the BLE peripheral: "
@@ -437,12 +482,14 @@ class BLESession(Session):
     def __del__(self):
         self.close()
 
-    def _get_dev_uuid(self, dev):
+    def _get_dev_uuids(self, dev):
         for adtype in self.SERVICE_CLASS_UUID_ADTYPES:
-            service_class_uuid = dev.getValueText(adtype)
-            if service_class_uuid:
-                logger.debug(self.SERVICE_CLASS_UUID_ADTYPES[adtype])
-                return UUID(service_class_uuid)
+            service_class_uuids = dev.getValue(adtype)
+            if service_class_uuids:
+                for u in service_class_uuids:
+                    a = self.SERVICE_CLASS_UUID_ADTYPES[adtype]
+                    logger.debug(f"service class uuid for {a}/{adtype}: {u}")
+                return service_class_uuids
         return None
 
     def matches(self, dev, filters):
@@ -455,18 +502,19 @@ class BLESession(Session):
                 for s in f['services']:
                     logger.debug(f"service to check: {s}")
                     given_uuid = s
-                    logger.debug(f"given: {given_uuid}")
-                    dev_uuid = self._get_dev_uuid(dev)
-                    if not dev_uuid:
+                    logger.debug(f"given UUID: {given_uuid} hash={UUID(given_uuid).__hash__()}")
+                    dev_uuids = self._get_dev_uuids(dev)
+                    if not dev_uuids:
                         continue
-                    logger.debug(f"dev: {dev_uuid}")
-                    logger.debug(given_uuid == dev_uuid)
-                    if given_uuid == dev_uuid:
+                    for u in dev_uuids:
+                        logger.debug(f"dev UUID: {u} hash={u.__hash__()}")
+                        logger.debug(given_uuid == u)
+                        if given_uuid == u:
                             logger.debug("match...")
                             return True
             if 'namePrefix' in f:
                 # 0x08: Shortened Local Name
-                deviceName = dev.getValueText(0x08)
+                deviceName = self._getDevName(dev)
                 if not deviceName:
                     continue
                 logger.debug(f"Name of \"{deviceName}\" begins with: \"{f['namePrefix']}\"?")
@@ -480,6 +528,31 @@ class BLESession(Session):
                 # ref: https://github.com/LLK/scratch-link/blob/develop/Documentation/BluetoothLE.md
         return False
 
+    def _scan_devices(self, params):
+        global scan_seconds
+        if BLESession.nr_connected > 0:
+            return len(BLESession.found_devices) > 0
+        found = False
+        with BLESession.scan_lock:
+            if not BLESession.scan_started:
+                BLESession.scan_started = True
+                BLESession.found_devices.clear()
+                for i in range(self.MAX_SCANNER_IF):
+                    scanner = Scanner(iface=i)
+                    try:
+                        logger.debug(f"start BLE scan: {scan_seconds} seconds")
+                        devices = scanner.scan(scan_seconds)
+                        for dev in devices:
+                            if self.matches(dev, params['filters']):
+                                BLESession.found_devices.append(dev)
+                                found = True
+                                logger.debug(f"BLE device found with iface #{i}");
+                    except BTLEManagementError as e:
+                        logger.debug(f"BLE iface #{i}: {e}");
+            else:
+                found = len(BLESession.found_devices) > 0
+        return found
+
     def _get_service(self, service_id):
         with self.lock:
             service = self.perip.getServiceByUUID(UUID(service_id))
@@ -528,29 +601,21 @@ class BLESession(Session):
         if self.status == self.INITIAL and method == 'discover':
             if not bluepy_helper_cap.is_set():
                 logger.error("Capability is not set to bluepy helper.")
-                logger.error("Run bluepy_setcap.py with root privilege.")
+                logger.error("Run bluepy_helper_cap(.py).")
+                logger.error("e.g. $ bluepy_helper_cap")
                 logger.error("e.g. $ sudo bluepy_helper_cap.py")
                 sys.exit(1)
-            found_ifaces = 0
-            for i in range(self.MAX_SCANNER_IF):
-                scanner = Scanner(iface=i)
-                try:
-                    devices = scanner.scan(10.0)
-                    for dev in devices:
-                        if self.matches(dev, params['filters']):
-                            self.found_devices.append(dev)
-                    found_ifaces += 1
-                    logger.debug(f"BLE device found with iface #{i}");
-                except BTLEManagementError as e:
-                    logger.debug(f"BLE iface #{i}: {e}");
-
-            if found_ifaces == 0:
+            found = self._scan_devices(params)
+            if not found:
+                if BLESession.nr_connected > 0:
+                    err_msg = "Can not scan BLE devices. Disconnect other sessions."
+                elif len(BLESession.found_devices) == 0:
                     err_msg = "Can not scan BLE devices. Check BLE controller."
                 logger.error(err_msg);
                 res["error"] = { "message": err_msg }
                 self.status = self.DONE
 
-            if len(self.found_devices) == 0 and not err_msg:
+            if len(BLESession.found_devices) == 0 and not err_msg:
                 err_msg = (f"No BLE device found: {params['filters']}. "
                            "Check BLE device.")
                 res["error"] = { "message": err_msg }
@@ -564,11 +629,12 @@ class BLESession(Session):
 
         elif self.status == self.DISCOVERY and method == 'connect':
             logger.debug("connecting to the BLE device")
-            self.device = self.found_devices[params['peripheralId']]
-            self.deviceName = self.device.getValueText(0x9) or self.device.getValueText(0x8)
+            self.device = BLESession.found_devices[params['peripheralId']]
+            self.deviceName = self._getDevName(self.device)
             try:
                 self.perip = Peripheral(self.device)
                 logger.info(f"connected to the BLE peripheral: {self.deviceName}")
+                BLESession.found_devices.remove(self.device)
             except BTLEDisconnectError as e:
                 logger.error(f"failed to connect to the BLE device \"{self.deviceName}\": {e}")
                 self.status = self.DONE
@@ -576,6 +642,8 @@ class BLESession(Session):
             if self.perip:
                 res["result"] = None
                 self.status = self.CONNECTED
+                BLESession.nr_connected += 1
+                logger.debug(f"BLE session connected={BLESession.nr_connected}")
                 self.delegate = self.BLEDelegate(self)
                 self.perip.withDelegate(self.delegate)
                 self._cache_characteristics()
@@ -639,11 +707,19 @@ class BLESession(Session):
         service = self._get_service(service_id)
         c = self._get_characteristic(chara_id)
         handle = c.getHandle()
+        # get CCCD or Client Characterstic Configuration Descriptor
+        cccd = None
+        for d in c.getDescriptors():
+            if d.uuid == UUID(CCCD_UUID):
+                cccd = d
+        if not cccd:
+            logger.error("Characteristic {char_id} does not have CCCD")
+            return
         # prepare notification handler
         self.delegate.add_handle(service_id, chara_id, handle)
         # request notification to the BLE device
         with self.lock:
-            self.perip.writeCharacteristic(handle + 1, value, True)
+            self.perip.writeCharacteristic(cccd.handle, value, True)
 
     def startNotifications(self, service_id, chara_id):
         logger.debug(f"start notification for {chara_id}")
@@ -668,7 +744,8 @@ async def ws_handler(websocket, path):
         await session.handle()
     except Exception as e:
         logger.error(f"Failure in session for web socket path: {path}")
-        logger.error(e)
+        logger.error(f"{type(e).__name__}: {e}")
+        session.close()
 
 def stack_trace():
     print("in stack_trace")
@@ -685,18 +762,20 @@ def stack_trace():
          print(line)
 
 def main():
-    opts = [opt for opt in sys.argv[1:] if opt.startswith("-")]
-    if "-h" in opts:
-        print((f"Usage: {sys.argv[0]} [OPTS]\n"
-               "OPTS:\t-h Show this help.\n"
-               "\t-d Print debug messages."
-        ))
-        sys.exit(1)
-    elif "-d" in opts:
+    global scan_seconds
+    parser = argparse.ArgumentParser(description='start Scratch-link')
+    parser.add_argument('-d', '--debug', action='store_true',
+                        help='print debug messages')
+    parser.add_argument('-s', '--scan_seconds', type=float, default=10.0,
+                        help='specifiy duration to scan BLE devices in seconds')
+    args = parser.parse_args()
+    if args.debug:
         print("Print debug messages")
         logLevel = logging.DEBUG
         handler.setLevel(logLevel)
         logger.setLevel(logLevel)
+    scan_seconds = args.scan_seconds
+    logger.debug(f"set scan_seconds: {scan_seconds}")
 
     # Prepare certificate of the WSS server
     gencert.prep_cert()
@@ -708,7 +787,7 @@ def main():
     ssl_context.load_cert_chain(localhost_cer, localhost_key)
 
     start_server = websockets.serve(
-         ws_handler, "device-manager.scratch.mit.edu", 20110, ssl=ssl_context
+         ws_handler, HOSTNAME, 20110, ssl=ssl_context
     )
 
     while True:
@@ -719,7 +798,13 @@ def main():
         except KeyboardInterrupt as e:
             stack_trace()
             break
+        except socket.gaierror as e:
+            logger.error(f"{type(e).__name__}: {e}")
+            logger.info(f"Check internet connection to {HOSTNAME}. If not "
+                        f"available, add '127.0.0.1 {HOSTNAME}' to /etc/hosts.")
+            break
         except Exception as e:
+            logger.error(f"{type(e).__name__}: {e}")
             logger.info("Restarting scratch-link...")
 
 if __name__ == "__main__":

setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setuptools.setup(
     name="pyscrlink",
-    version="0.2.0",
+    version="0.2.4",
     author="Shin'ichiro Kawasaki",
     author_email='kawasaki@juno.dti.ne.jp',
     description='Scratch-link for Linux with Python',