ble: implement BLEDBusSession class

Current implementation of BLESession class depends on the library bluepy
to communicate with BLE devices. This implementation has two major
issues. The first issue is no maintenance of bluepy library. It's last
commit was made in May 2021. It is no longer active and it is not likely
that its issues get resolved. The second issue is lack of asyncio
interface. pyscrlink uses websockets which uses asyncio. Dirty glue code
and locks are required to connect websockets and bluepy.

This commit avoids the issue by moving away from bluepy. Instead, use
BlueZ D-Bus API. Bluepy communicates with BlueZ kernel module via bluepy
unique user space program. Instead, pyscrlink communicate with well
maintained Blue-Z user space program, or bluetoothd, through D-Bus
protocol. To handle D-Bus protocol, use python-sdbus library [2]. It has
sub-library python-sdbus-bluez to cover BlueZ D-Bus API [3].

Using these libraries, implement a new class named BLEDBusSession. It
replaces BLESession which uses bluepy. These two classes share same
base class named Session. To allow async I/O, add a new function
async_handle_request to the Session class. Until this new implementation
get proved stable enough, keep the old BLESession. Introduce a new
command line option '-b' of scratch_link. When this option is specified,
the new class is used.

To improve code readability, implement BLEDBusSession in a new file
'ble.py'. To share scan_seconds between scratch_link.py and ble.py,
change scan_seconds from global variable to an argument of Session class
constructor. Also to share logger between scratch_link.py and ble.py,
modify the logging.getLogger() argument from __name__ to scratch_link
module name.

[1] http://www.bluez.org/bluez-5-api-introduction-and-porting-guide/
[2] https://github.com/python-sdbus/python-sdbus
[3] https://github.com/python-sdbus/python-sdbus-bluez

Signed-off-by: Shin'ichiro Kawasaki <kawasaki@juno.dti.ne.jp>

.gitignore

@@ -1 +1,5 @@
-/__pycache__
+__pycache__
+build
+dist
+pyscrlink/__pycache__
+pyscrlink.egg-info

README.md

@@ -9,11 +9,16 @@ micro:bit.
 
 Pyscrlink allows you to connect Scratch and bluetooth devices with the Linux
 OSes. It uses the Linux Bluetooth protocol stack [Bluez](http://www.bluez.org/)
-and its python interfaces [pybluez](https://github.com/pybluez/pybluez) to
-handle Bluetooth, and [bluepy](https://github.com/IanHarvey/bluepy) to handle
-Bluetooth Low Energy (BLE) connections. It has been reported that pyscrlink
-connects Scratch 3.0 with micro:bit, LEGO Mindstorms EV3, LEGO WeDo, LEGO
-Boost, Intelino Smart Train and toio.
+and [bluepy](https://github.com/IanHarvey/bluepy) to handle Bluetooth Low Energy
+(BLE) connections. It has been reported that pyscrlink connects Scratch 3.0 with
+micro:bit, LEGO WeDo, LEGO Boost and toio.
+
+Until version v0.2.5, pyscrlink supported Bluetooth Classic protocol using
+[pybluez](https://github.com/pybluez/pybluez). Unfortunately, pybluez is not
+well maintained and caused technical troubles. Then Bluetooth Classic protocol
+support is dropped from pyscrlink. This means that LEGO Mindstorm EV3 can not
+be connected with pyscrlink. Bluetooth Classic support is the improvement
+opportunity of pyscrlink.
 
 To use websockets, pyscrlink requires python version 3.6 or later. If your
 system has python older than version 3.6, install newer version. If your
@@ -39,26 +44,24 @@ Devices:
 
 Linux distros:
 * Arch Linux
-* elementary OS 5.1 Hera
 
 Browsers:
-* FireFox
+* Firefox
 * Chromium
 
 It was reported that pyscrlink (former bluepy-scratch-link) working with
 following devices and Linux distros.
 
 Devices:
-* LEGO Mindstorm EV3 by @chrisglencross
 * LEGO WeDo by @zhaowe, @KingBBQ
-* LEGO Boost by @laurentchar, @miguev, @jacquesdt
-* Intelino Smart Train by @ErrorJan
+* LEGO Boost and compatible devices by @laurentchar, @miguev, @jacquesdt, @n3storm
 * toio by @shimodash
 
 Linux distros:
 * Raspbian by @chrisglencross
 * Ubuntu 16.04 @jacquesdt
 * Ubuntu Studio 20.04 @miguev
+* Debian 11 @n3storm
 
 Installation
 ------------
@@ -87,7 +90,7 @@ Installation
     $ pip3 install pyscrlink
     ```
 
-4. For Bluetooth Low Energy (BLE) devices, set bluepy-helper capability.
+4. Set bluepy-helper capability.
 
     ```
     $ bluepy_helper_cap
@@ -99,72 +102,25 @@ Installation
 
 5. For micro:bit, install Scratch-link hex on your device.
 
-    * Download and unzip the [micro:bit Scratch Hex file](https://downloads.scratch.mit.edu/microbit/scratch-microbit-1.1.0.hex.zip).
+    * Download and unzip the [micro:bit Scratch Hex file](https://downloads.scratch.mit.edu/microbit/scratch-microbit.hex.zip).
     * Flash the micro:bit over USB with the Scratch Hex File, you will see the
       five character name of the micro:bit scroll across the screen such as
       'zo9ev'.
 
 Usage
 -----
-1. For LEGO Mindstorms EV3, pair your Linux PC to the EV3 brick.
-
-   First, turn on the EV3 and ensure Bluetooth is enabled.
-
-   Then, pair using your Linux desktop's the Bluetooth settings.
-
-   If using Gnome:
-      * Settings -> Bluetooth
-      * Click on the EV3 device name
-      * Accept the connection on EV3 brick
-      * Enter a matching PIN on EV3 brick and Linux PC. '1234' is the value Scratch suggests.
-      * Confirm EV3 status is "Disconnected" in Bluetooth settings
-
-   With a Raspberry Pi default Raspbian desktop, click the Bluetooth logo in the top right of the screen and
-   Add Device. Then follow the Gnome instructions. You will be warned that the Raspberry Pi
-   does not know how to talk to this device; that is not a problem.
-
-   Alternatively you can perform pairing from the command-line:
-   ```shell script
-   $ bluetoothctl
-
-   [bluetooth]# power on
-   Changing power on succeeded
-
-   [bluetooth]# pairable on
-   Changing pairable on succeeded
-
-   [bluetooth]# agent KeyboardOnly
-   Agent registered
-
-   [bluetooth]# devices
-   ...
-   Device 00:16:53:53:D3:19 EV3
-   ...
-
-   [bluetooth]# pair 00:16:53:53:D3:19
-   Attempting to pair with 00:16:53:53:D3:19
-
-   # Confirm pairing on the EV3 display, set PIN to 1234
-
-   Request PIN code
-   [agent] Enter PIN code: 1234
-   [CHG] Device 00:16:53:53:D3:19 Connected: yes
-   [CHG] Device 00:16:53:53:D3:19 Paired: yes
-   Pairing successful
-
-   [bluetooth]# quit
-   ```
-
-2. Start scratch-link python script.
+1. Start scratch-link python script.
     ```sh
     $ scratch_link
     ```
+    If your device is toio, add "-s 1" option to the scratch_link command. It
+    allows the toio Do Visual Programming to connect to toio automatically.
 
-3. Connect scratch to the target device such as micro:bit or LEGO Mindstorms:
+2. Connect scratch to the target device such as micro:bit:
     * Open FireFox or Chrome. (Make sure to run as the same user for scratch-link python script.)
     * Access [Scratch 3.0](https://scratch.mit.edu/) and create your project.
     * Select the "Add Extension" button.
-    * Select the extension for your device (e.g., micro:bit or Lego Mindstorms EV3 extension) and follow the prompts to connect.
+    * Select the extension for your device (e.g., micro:bit) and follow the prompts to connect.
     * Build your project with the extension blocks.
 
 In Case You Fail to Connect
@@ -185,10 +141,22 @@ In Case You Fail to Connect
     * If the service is not working, refer guide of your distro to set it up.
     * If the service is working, also check that /etc/bluetooth/main.conf sets AutoEnable=true.
 
-3. If scratch_link.py says "failed to connect to BT device: [Errno 13] Permission denied",
+3. If device scan still fails, use -r option to retry device scan.
+    The command line below does device scan twice. Each scan takes 10 seconds.
+    ```
+    $ scratch_link -r 2
+    ```
+    It would be good to use -s option together to reduce each scan duration.
+    The command line below does 3 seconds device scan twice.
+
+    ```
+    $ scratch_link -r 2 -s 3
+    ```
+
+4. If scratch_link.py says "failed to connect to BT device: [Errno 13] Permission denied",
    make sure to pair the bluetooth device to your PC before connecting to Scratch.
 
-4. To connect to multiple devices at the same time, make all the target devices
+5. To connect to multiple devices at the same time, make all the target devices
    ready for scan at the first device scan. This is important for toio. The toio
    allows a single project to connect to two toio devices.
    * When the second device was prepared after the first device was connected, device scan can not find the second device.
@@ -202,6 +170,32 @@ Please file issues to [GitHub issue tracker](https://github.com/kawasaki/pyscrli
 Releases
 --------
 
+Release 0.2.8
+
+* Supported Microbit More v2
+
+Release 0.2.7
+
+* Supported Snap Firefox and Chromium
+* Added -r option to retry BLE scan
+
+Release 0.2.6
+
+* Removed Bluetooth Classic and LEGO Mindstorm EV3 support
+
+Release 0.2.5
+
+* Fixed handling of multiple UUIDs for LEGO Boost
+
+Release 0.2.4
+
+* Added -s option to specify BLE scan duration
+* Improved README.md
+
+Release 0.2.3
+
+* Fixed eternal loop caused by hostname resolve failure
+
 Release 0.2.2
 
 * Supported multiple device connections for toio

pyscrlink/ble.py

@@ -0,0 +1,374 @@
+from sdbus import DbusInterfaceCommonAsync, SdBus, sd_bus_open_system
+from sdbus.dbus_proxy_async_interfaces import DbusIntrospectableAsync
+import xml.etree.ElementTree as ET
+from sdbus_async.bluez.adapter_api import AdapterInterfaceAsync
+from sdbus_async.bluez.device_api import DeviceInterfaceAsync
+from sdbus_async.bluez.gatt_api import (
+    GattCharacteristicInterfaceAsync,
+    GattServiceInterfaceAsync,
+)
+import asyncio
+import base64
+from asyncio import sleep
+from os import dup, fdopen, close
+
+import pyscrlink.scratch_link
+from pyscrlink.scratch_link import BTUUID
+
+import logging
+logger = logging.getLogger('pyscrlink.scratch_link')
+
+class BLEDBusSession(pyscrlink.scratch_link.Session):
+    """
+    Manage a session for Bluetooth Low Energy device such as micro:bit using
+    DBus as backend.
+    """
+
+    INITIAL = 1
+    DISCOVERY = 2
+    CONNECTED = 3
+    DONE = 4
+
+    MAX_SCANNER_IF = 3
+
+    connected_devices = {}
+
+    class Device():
+        def __init__(self, iface, path, node_name, name, address):
+            self.iface = iface
+            self.path = path
+            self.node_name = node_name
+            self.name = name
+            self.address = address
+
+    class Notification():
+        def __init__(self, loop, acquired_fd, fd, fp, params):
+            self.loop = loop
+            self.acquired_fd = acquired_fd
+            self.fd = fd
+            self.fp = fp
+            self.params = params
+
+        def close(self):
+            self.loop.remove_reader(self.fd)
+            self.fp.close()
+
+    def _connect_to_adapters(self):
+        self.iface = None
+        self.adapter = None
+        self.adapter_introspect = None
+        adapter = AdapterInterfaceAsync()
+        for i in range(self.MAX_SCANNER_IF):
+            iface = '/org/bluez/hci' + str(i)
+            logger.debug(f"try connect to {iface}")
+            try:
+                adapter._connect('org.bluez', iface, bus=self.dbus)
+                logger.debug(f"connected to {iface}")
+                adapter_introspect = DbusIntrospectableAsync()
+                adapter_introspect._connect('org.bluez', iface, bus=self.dbus)
+                self.iface = iface
+                self.adapter = adapter
+                self.adapter_introspect = adapter_introspect
+                return
+            except Exception as e:
+                logger.error(e)
+        raise Exception("no adapter is available")
+
+    async def _start_discovery(self):
+        logger.debug(f"Starting discovery... {self.adapter}")
+        assert not self.discovery_running
+        await self.adapter.start_discovery()
+        self.discovery_running = True
+
+        asyncio.create_task(self._find_devices())
+        asyncio.create_task(self._stop_discovery())
+        logger.debug(f"Task to stop discovery has got created.")
+
+    async def _matches(self, dev, filters):
+        """
+        Check if the found BLE device matches the filters Scratch specifies.
+        """
+        logger.debug(f"in matches {dev} {filters}")
+        for f in filters:
+            if 'services' in f:
+                for s in f['services']:
+                    logger.debug(f"service to check: {s}")
+                    given_uuid = BTUUID(s)
+                    logger.debug(f"given UUID: {given_uuid} hash={given_uuid.__hash__()}")
+                    dev_uuids = await dev.iface.uuids
+                    if not dev_uuids:
+                        logger.debug(f"dev UUID not available")
+                        continue
+                    for uuid in dev_uuids:
+                        u = BTUUID(uuid)
+                        logger.debug(f"dev UUID: {u} hash={u.__hash__()}")
+                        logger.debug(given_uuid == u)
+                        if given_uuid == u:
+                            logger.debug("match...")
+                            return True
+            if 'namePrefix' in f:
+                logger.debug(f"given namePrefix: {f['namePrefix']}")
+                if dev.name:
+                    logger.debug(f"name: {dev. name}")
+                    if dev.name.startswith(f['namePrefix']):
+                        logger.debug(f"match...")
+                        return True
+            if 'name' in f or 'manufactureData' in f:
+                logger.error("name/manufactureData filters not implemented")
+                # TODO: implement other filters defined:
+                # ref: https://github.com/LLK/scratch-link/blob/develop/Documentation/BluetoothLE.md
+        return False
+
+    async def _notify_device(self, device) -> None:
+        params = { 'rssi': -80, 'name': 'Unknown' }
+        try:
+            params['rssi'] = await device.iface.rssi
+        except Exception:
+            None
+        if device.name:
+            params['name'] = device.name
+        params['peripheralId'] = device.node_name
+        await self._send_notification('didDiscoverPeripheral', params)
+
+    async def _find_devices(self) -> None:
+        assert self.discovery_running
+        while self.discovery_running:
+            await sleep(1)
+            s = await self.adapter_introspect.dbus_introspect()
+            parser = ET.fromstring(s)
+            nodes = parser.findall("./node")
+            if not nodes:
+                logger.info("device not found")
+                continue
+            logger.debug(f"{len(nodes)} device(s) found")
+            for node in nodes:
+                node_name = node.attrib['name']
+                logger.debug(f"  {node_name}")
+                if self.found_devices.get(node_name):
+                    continue
+                devpath = self.iface + "/" + node_name
+                if BLEDBusSession.connected_devices.get(devpath):
+                    continue
+                iface = DeviceInterfaceAsync()
+                iface._connect('org.bluez', devpath, bus=self.dbus)
+                try:
+                    devname = await iface.name
+                except Exception as e:
+                    logger.debug(f"device {node_name} does not have name: {e}")
+                devaddr = await iface.address
+                device = self.Device(iface, devpath, node_name, devname,
+                                     devaddr)
+                if not await self._matches(device, self.discover_filters):
+                    await iface.disconnect()
+                    continue
+                self.found_devices[node_name] = device
+                await self._notify_device(device)
+
+        logger.debug("end _find_device.")
+
+    async def _stop_discovery(self) -> None:
+        assert self.discovery_running
+        logger.debug(f"Wait discovery for {self.scan_seconds} seconds")
+        await sleep(self.scan_seconds)
+        logger.debug(f"Stopping discovery... {self.adapter}")
+        self.discovery_running = False
+        await self.adapter.stop_discovery()
+
+    def __init__(self, websocket, loop, scan_seconds):
+        super().__init__(websocket, loop, scan_seconds)
+        logger.debug("dbus init")
+        self.status = self.INITIAL
+        self.dbus = sd_bus_open_system()
+        self.discovery_running = False
+        self.iface = None
+        self.services = {}
+        self.chars = {}
+        self.chars_cache = {}
+        self.notifications = {}
+        self._connect_to_adapters()
+        self.found_devices = {}
+
+    async def _get_characteristics(self, service_path):
+        service_introspect = DbusInterfaceCommonAsync()
+        service_introspect._connect('org.bluez', service_path, bus=self.dbus)
+        s = await service_introspect.dbus_introspect()
+        parser = ET.fromstring(s)
+        nodes = parser.findall("./node")
+        if not nodes:
+            logger.error(f"characteristic not found at {service_path}")
+            return
+        for node in nodes:
+            path = service_path + '/' + node.attrib['name']
+            if self.chars.get(path):
+                continue
+            logger.debug(f"getting GATT characteristic at {path}")
+            char = GattCharacteristicInterfaceAsync()
+            char._connect('org.bluez', path, bus=self.dbus)
+            self.chars[path] = char
+            cid = await char.uuid
+            logger.debug(f"found char {cid}")
+
+    async def _get_services(self):
+        # do D-Bus introspect to the device path and get service paths under it
+        for i in range(5):
+            dev_introspect = DbusInterfaceCommonAsync()
+            dev_introspect._connect('org.bluez', self.device.path,
+                                    bus=self.dbus)
+            s = await dev_introspect.dbus_introspect()
+            parser = ET.fromstring(s)
+            nodes = parser.findall("./node")
+            if nodes:
+                break
+            else:
+                logger.error("Service not found. Try again.")
+                await sleep(1)
+        if not nodes:
+            return []
+        for node in nodes:
+            path = self.device.path + '/' + node.attrib['name']
+            if self.services.get(path):
+                continue
+            logger.debug(f"getting GATT service at {path}")
+            service = GattServiceInterfaceAsync()
+            service._connect('org.bluez', path, bus=self.dbus)
+            self.services[path] = service
+            sid = await service.uuid
+            logger.debug(f"found service {sid}")
+            await self._get_characteristics(path)
+
+    async def _get_char(self, id):
+        char = self.chars_cache.get(id)
+        if char:
+            return char
+        for i in range(5):
+            await self._get_services()
+            btuuid = BTUUID(id)
+            for char in self.chars.values():
+                raw_uuid = await char.uuid
+                char_uuid = BTUUID(raw_uuid)
+                if char_uuid == btuuid:
+                    self.chars_cache[id] = char
+                    return char
+            logger.error(f"Can not get characteristic: {id}. Retry.")
+        logger.error(f"Abandoned to get characteristic: {id}.")
+        return None
+
+    async def _start_notification(self, sid, cid, char):
+        logger.debug('startNotification')
+        (acquired_fd, mtu) = await char.acquire_notify({})
+        fd = dup(acquired_fd)
+        fp = fdopen(fd, mode='rb', buffering=0, newline=None)
+        self.loop.add_reader(fd, self._read_notification, fd)
+        notification = self.Notification(self.loop, acquired_fd, fd, fp, {
+            'serviceId': sid,
+            'characteristicId': cid,
+            'encoding': 'base64'
+        })
+        self.notifications[fd] = notification
+        logger.debug(f'added notification reader: {notification}')
+
+    def _stop_notifications(self):
+        for n in self.notifications.values():
+            n.close()
+
+    def _read_notification(self, *args):
+        fd = args[0]
+        notification = self.notifications[fd]
+        data = notification.fp.read()
+        if len(data) == 0:
+            logger.debug(f'empty notification data')
+            asyncio.create_task(self.async_close())
+            return
+        params = notification.params.copy()
+        params['message'] = base64.standard_b64encode(data).decode('ascii')
+        self.loop.create_task(self._send_notification('characteristicDidChange', params))
+
+    def handle_request(self, method, params):
+        logger.debug("handle request")
+
+    async def async_handle_request(self, method, params):
+        logger.debug(f"async handle request: {method} {params}")
+
+        res = { "jsonrpc": "2.0" }
+        err_msg = None
+
+        if self.status == self.INITIAL and method == 'discover':
+            self.discover_filters = params['filters']
+            logger.debug(f"discover: {self.discover_filters}")
+            try:
+                await self._start_discovery()
+                logger.debug(f"discover started: {self.discover_filters}")
+                res["result"] = None
+                self.status = self.DISCOVERY
+            except Exception as e:
+                res["error"] = { "message": "Failed to start device discovery" }
+                self.status = self.DONE
+
+        elif self.status == self.DISCOVERY and method == 'connect':
+            logger.debug("connecting to the BLE device")
+            dev = self.found_devices[params['peripheralId']]
+            try:
+                logger.debug(f"  {dev}")
+                await dev.iface.connect()
+                res["result"] = None
+                self.device = dev
+                self.status = self.CONNECTED
+                logger.info(f"Connected: '{dev.name}'@{dev.address}")
+                BLEDBusSession.connected_devices[dev.path] = dev
+            except NotImplementedError as e:
+                logger.error(e)
+                res["error"] = { "message": "Failed to connect to device" }
+                self.status = self.DONE
+            except Exception as e:
+                logger.error(f"failed to connect: {e}")
+                res["error"] = { "message": "Failed to connect to device" }
+                self.status = self.DONE
+
+        elif self.status == self.CONNECTED and method == 'read':
+            logger.debug("handle read request")
+            service_id = params['serviceId']
+            chara_id = params['characteristicId']
+            c = await self._get_char(chara_id)
+            value = await c.read_value({})
+            message = base64.standard_b64encode(value).decode('ascii')
+            res['result'] = { 'message': message, 'encode': 'base64' }
+            if params.get('startNotifications') == True:
+                await self._start_notification(service_id, chara_id, c)
+
+        elif self.status == self.CONNECTED and method == 'write':
+            logger.debug(f"handle write request {params}")
+            service_id = params['serviceId']
+            chara_id = params['characteristicId']
+            c = await self._get_char(chara_id)
+            if params['encoding'] != 'base64':
+                logger.error("encoding other than base 64 is not "
+                             "yet supported: ", params['encoding'])
+            else:
+                msg_bstr = params['message'].encode('ascii')
+                data = base64.standard_b64decode(msg_bstr)
+                await c.write_value(data, {})
+                res['result'] = len(data)
+
+        logger.debug(res)
+        return res
+
+    def end_request(self):
+        logger.debug("end request")
+        return False
+
+    async def async_close(self):
+        if not self.device:
+            return
+        dev = self.device
+        logger.info(f"Disconnecting from '{dev.name}'@{dev.address}")
+        self._stop_notifications()
+        await dev.iface.disconnect()
+        BLEDBusSession.connected_devices.pop(dev.path)
+        logger.info(f"Disconnected from '{dev.name}'@{dev.address}")
+        self.device = None
+        await self.websocket.close()
+        return
+
+    def close(self):
+        logger.debug("close")
+        return

pyscrlink/gencert.py

@@ -152,36 +152,37 @@ def prep_nss_cert(dir, cert, nickname):
     remove_cert(dir, nickname)
     add_cert(dir, cert, nickname)
 
-def prep_cert():
-    # Generate certification and key
-    gen_cert(cert_file_path, key_file_path)
-
-    # Add certificate to FireFox
+def prep_cert_for_app(cert, app, search_path):
+    """
+    Find a NSS DB in the search_path for the app and prepare the cert in the DB.
+    """
     nssdb = None
-    firefox_nss_path = os.path.join(homedir, ".mozilla/firefox/")
-    for root, dirs, files in os.walk(firefox_nss_path):
+    for root, dirs, files in os.walk(os.path.join(homedir, search_path)):
         for name in files:
             if not re.match("key.*\.db", name):
                 continue
             nssdb = root
-            if prep_nss_cert(nssdb, cert_file_path, SCRATCH_CERT_NICKNAME):
-                logger.error(f"Failed to add certificate to FireFox NSS DB: {nssdb}")
+            if prep_nss_cert(nssdb, cert, SCRATCH_CERT_NICKNAME):
+                logger.error(f"Failed to add certificate to {app}: {nssdb}")
                 sys.exit(3)
             else:
-                logger.info(f"Certificate is ready in FireFox NSS DB: {nssdb}")
+                logger.info(f"Certificate is ready in {app} NSS DB: {nssdb}")
     if not nssdb:
-        logger.info("FireFox NSS DB not found. Do not add certificate.")
+        logger.debug(f"NSS DB for {app} not found. Do not add certificate.")
 
-    # Add certificate to Chrome
-    nssdb = os.path.join(homedir, ".pki/nssdb")
-    if os.path.isdir(nssdb):
-        if prep_nss_cert(nssdb, cert_file_path, SCRATCH_CERT_NICKNAME):
-            logger.error(f"Failed to add certificate to Chrome")
-            sys.exit(4)
-        else:
-            logger.info("Certificate is ready for Chrome")
-    else:
-        logger.info("Chrome NSS DB not found. Do not add certificate.")
+
+def prep_cert():
+    # Generate certification and key
+    gen_cert(cert_file_path, key_file_path)
+
+    nss_dbs = {
+        "FireFox": ".mozilla/firefox/",
+        "FireFox(Snap)": "snap/firefox/common/.mozilla/firefox/",
+        "Chrome": ".pki",
+        "Chromium(Snap)": "snap/chromium",
+        }
+
+    [ prep_cert_for_app(cert_file_path, k, nss_dbs[k]) for k in nss_dbs ]
 
 if __name__ == "__main__":
     prep_cert()

pyscrlink/scratch_link.py

@@ -8,45 +8,61 @@ import asyncio
 import pathlib
 import ssl
 import websockets
+import socket
 import json
+import uuid
 import base64
 import logging
 import sys
 import signal
 import traceback
-
-# for Bluetooth (e.g. Lego EV3)
-import bluetooth
+import argparse
 
 # for BLESession (e.g. BBC micro:bit)
-from bluepy.btle import Scanner, UUID, Peripheral, DefaultDelegate
+from bluepy.btle import Scanner, UUID, Peripheral, DefaultDelegate, ScanEntry
 from bluepy.btle import BTLEDisconnectError, BTLEManagementError
 from pyscrlink import bluepy_helper_cap
 
 import threading
 import time
 import queue
+from asyncio import sleep
 
 # for websockets certificate
 from pyscrlink import gencert
 
-logLevel = logging.INFO
+from pyscrlink import ble
 
 # for logging
-logger = logging.getLogger(__name__)
-formatter = logging.Formatter(fmt='%(asctime)s %(message)s')
-handler = logging.StreamHandler()
-handler.setLevel(logLevel)
-handler.setFormatter(formatter)
-logger.setLevel(logLevel)
-logger.addHandler(handler)
-logger.propagate = False
+logger = logging.getLogger('pyscrlink.scratch_link')
+
+HOSTNAME="device-manager.scratch.mit.edu"
+scan_seconds=10.0
+
+class BTUUID(uuid.UUID):
+    BLUETOOTH_BASE_UUID = "00001000800000805F9B34FB"
+
+    def __init__(self, val):
+        if isinstance(val, int):
+            if (val < 0) or (val > 0xFFFFFFFF):
+                raise ValueError(
+                    "Short form UUIDs must be in range 0..0xFFFFFFFF")
+            val = "%04X" % val
+        else:
+            val = str(val)
+
+        val = val.replace("-", "")
+        if len(val) <= 8:  # Short form
+            val = ("0" * (8 - len(val))) + val + self.BLUETOOTH_BASE_UUID
+
+        uuid.UUID.__init__(self, val)
 
 class Session():
     """Base class for BTSession and BLESession"""
-    def __init__(self, websocket, loop):
+    def __init__(self, websocket, loop, scan_seconds):
         self.websocket = websocket
         self.loop = loop
+        self.scan_seconds = scan_seconds
         self.lock = threading.RLock()
         self.notification_queue = queue.Queue()
 
@@ -65,7 +81,11 @@ class Session():
         if jsonreq['jsonrpc'] != '2.0':
             logger.error("error: jsonrpc version is not 2.0")
             return True
-        jsonres = self.handle_request(jsonreq['method'], jsonreq['params'])
+        if type(self) is ble.BLEDBusSession:
+            jsonres = await self.async_handle_request(jsonreq['method'],
+                                                      jsonreq['params'])
+        else:
+            jsonres = self.handle_request(jsonreq['method'], jsonreq['params'])
         if 'id' in jsonreq:
             jsonres['id'] = jsonreq['id']
         response = json.dumps(jsonres)
@@ -79,6 +99,10 @@ class Session():
         """Default request handler"""
         logger.debug(f"default handle_request: {method}, {params}")
 
+    async def async_handle_request(self, method, params):
+        """Default async request handler"""
+        logger.debug(f"default async handle_request: {method}, {params}")
+
     def end_request(self):
         """
         Default callback at request end. This callback is required to
@@ -118,11 +142,17 @@ class Session():
                     break
                 await self._send_notifications()
                 logger.debug("in handle loop")
-            except websockets.ConnectionClosedError as e:
+            except (websockets.ConnectionClosedOK, websockets.ConnectionClosedError) as e:
                 logger.info("scratch closed session")
                 logger.debug(e)
-                self.close()
+                if type(self) is ble.BLEDBusSession:
+                    await self.async_close()
+                else:
+                    self.close()
                 break
+            except Exception as e:
+                t = type(e)
+                logger.info(f"scratch closed with unkown exception: {e}: {t}")
 
     def close(self):
         """
@@ -133,198 +163,10 @@ class Session():
 class BTSession(Session):
     """Manage a session for Bluetooth device"""
 
-    INITIAL = 1
-    DISCOVERY = 2
-    DISCOVERY_COMPLETE = 3
-    CONNECTED = 4
-    DONE = 5
-
-    # Split this into discovery thread and communication thread
-    # discovery thread should auto-terminate
-
-    class BTThread(threading.Thread):
-        """
-        Separated thread to control notifications to Scratch.
-        It handles device discovery notification in DISCOVERY status
-        and notifications from bluetooth devices in CONNECTED status.
-        """
-
-        class BTDiscoverer(bluetooth.DeviceDiscoverer):
-
-            def __init__(self, major_class, minor_class):
-                super().__init__()
-                self.major_class = major_class
-                self.minor_class = minor_class
-                self.found_devices = {}
-                self.done = False
-
-            def pre_inquiry(self):
-                self.done = False
-
-            def device_discovered(self, address, device_class, rssi, name):
-                logger.debug(f"Found device {name} addr={address} class={device_class} rssi={rssi}")
-                major_class = (device_class & 0x1F00) >> 8
-                minor_class = (device_class & 0xFF) >> 2
-                if major_class == self.major_class and minor_class == self.minor_class:
-                    self.found_devices[address] = (name, device_class, rssi)
-
-            def inquiry_complete(self):
-                self.done = True
-
-        def __init__(self, session, major_device_class, minor_device_class):
-            threading.Thread.__init__(self)
-            self.session = session
-            self.major_device_class = major_device_class
-            self.minor_device_class = minor_device_class
-            self.cancel_discovery = False
-            self.ping_time = None
-
-        def discover(self):
-            discoverer = self.BTDiscoverer(self.major_device_class, self.minor_device_class)
-            discoverer.find_devices(lookup_names=True)
-            while self.session.status == self.session.DISCOVERY and not discoverer.done and not self.cancel_discovery:
-                readable = select.select([discoverer], [], [], 0.5)[0]
-                if discoverer in readable:
-                    discoverer.process_event()
-                    for addr, (device_name, device_class, rssi) in discoverer.found_devices.items():
-                        logger.debug(f"notifying discovered {addr}: {device_name}")
-                        params = {"rssi": rssi, 'peripheralId': addr, 'name': device_name.decode("utf-8")}
-                        self.session.notify('didDiscoverPeripheral', params)
-                    discoverer.found_devices.clear()
-
-            if not discoverer.done:
-                discoverer.cancel_inquiry()
-
-        def run(self):
-            while self.session.status != self.session.DONE:
-
-                logger.debug("loop in BT thread")
-                current_time = int(round(time.time()))
-
-                if self.session.status == self.session.DISCOVERY and not self.cancel_discovery:
-                    logger.debug("in discovery status:")
-                    try:
-                        self.discover()
-                        self.ping_time = current_time + 5
-                    finally:
-                        self.session.status = self.session.DISCOVERY_COMPLETE
-
-                elif self.session.status == self.session.CONNECTED:
-                    logger.debug("in connected status:")
-                    sock = self.session.sock
-                    try:
-                        ready = select.select([sock], [], [], 1)
-                        if ready[0]:
-                            header = sock.recv(2)
-                            [msg_len] = struct.unpack("<H", header)
-                            msg_data = sock.recv(msg_len)
-                            data = header + msg_data
-                            params = {'message': base64.standard_b64encode(data).decode('utf-8'), "encoding": "base64"}
-                            self.session.notify('didReceiveMessage', params)
-                            self.ping_time = current_time + 5
-
-                    except Exception as e:
-                            logger.error(e)
-                            self.session.close()
-                            break
-
-                    # To avoid repeated lock by this single thread,
-                    # yield CPU to other lock waiting threads.
-                    time.sleep(0)
-                else:
-                    # Nothing to do:
-                    time.sleep(1)
-
-                # Terminate if we have lost websocket connection to Scratch (e.g. browser closed)
-                if self.ping_time is None or self.ping_time <= current_time:
-                    try:
-                        self.session.notify('ping', {})
-                        self.ping_time = current_time + 5
-                    except Exception as e:
-                        logger.error(e)
-                        self.session.close()
-                        break
-
-    def __init__(self, websocket, loop):
-        super().__init__(websocket, loop)
-        self.status = self.INITIAL
-        self.sock = None
-        self.bt_thread = None
-
-    def close(self):
-        self.status = self.DONE
-        if self.sock:
-            logger.info(f"disconnect to BT socket: {self.sock}")
-            self.sock.close()
-            self.sock = None
-
-    def __del__(self):
-        self.close()
-
-    def handle_request(self, method, params):
-        """Handle requests from Scratch"""
-        logger.debug("handle request to BT device")
-        logger.debug(method)
-        if len(params) > 0:
-            logger.debug(params)
-
-        res = { "jsonrpc": "2.0" }
-
-        if self.status == self.INITIAL and method == 'discover':
-            logger.debug("Starting async discovery")
-            self.status = self.DISCOVERY
-            self.bt_thread = self.BTThread(self, params["majorDeviceClass"], params["minorDeviceClass"])
-            self.bt_thread.start()
-            res["result"] = None
-
-        elif self.status in [self.DISCOVERY, self.DISCOVERY_COMPLETE] and method == 'connect':
-
-            # Cancel discovery
-            while self.status == self.DISCOVERY:
-                logger.debug("Cancelling discovery")
-                self.bt_thread.cancel_discovery = True
-                time.sleep(1)
-
-            addr = params['peripheralId']
-            logger.debug(f"connecting to the BT device {addr}")
-            try:
-                self.sock = bluetooth.BluetoothSocket(bluetooth.RFCOMM)
-                self.sock.connect((addr, 1))
-                logger.info(f"connected to BT device: {addr}")
-            except bluetooth.BluetoothError as e:
-                logger.error(f"failed to connect to BT device: {e}", exc_info=e)
-                self.status = self.DONE
-                self.sock = None
-
-            if self.sock:
-                res["result"] = None
-                self.status = self.CONNECTED
-            else:
-                err_msg = f"BT connect failed: {addr}"
-                res["error"] = { "message": err_msg }
-                self.status = self.DONE
-
-        elif self.status == self.CONNECTED and method == 'send':
-            logger.debug("handle send request")
-            if params['encoding'] != 'base64':
-                logger.error("encoding other than base 64 is not "
-                                 "yet supported: ", params['encoding'])
-            msg_bstr = params['message'].encode('ascii')
-            data = base64.standard_b64decode(msg_bstr)
-            self.sock.send(data)
-            res['result'] = len(data)
-
-        logger.debug(res)
-        return res
-
-    def end_request(self):
-        logger.debug(f"end_request of BTSession {self}")
-        return self.status == self.DONE
-
-
 class BLESession(Session):
     """
-    Manage a session for Bluetooth Low Energy device such as micro:bit
+    Manage a session for Bluetooth Low Energy device such as micro:bit using
+    bluepy as backend.
     """
 
     INITIAL = 1
@@ -372,6 +214,7 @@ class BLESession(Session):
                     time.sleep(1)
                 elif self.session.status == self.session.CONNECTED:
                     logger.debug("in connected status:")
+
                     delegate = self.session.delegate
                     if delegate and len(delegate.handles) > 0:
                         if not delegate.restart_notification_event.is_set():
@@ -420,8 +263,8 @@ class BLESession(Session):
             params['message'] = base64.standard_b64encode(data).decode('ascii')
             self.session.notify('characteristicDidChange', params)
 
-    def __init__(self, websocket, loop):
-        super().__init__(websocket, loop)
+    def __init__(self, websocket, loop, scan_seconds):
+        super().__init__(websocket, loop, scan_seconds)
         self.status = self.INITIAL
         self.device = None
         self.deviceName = None
@@ -448,43 +291,50 @@ class BLESession(Session):
     def __del__(self):
         self.close()
 
-    def _get_dev_uuid(self, dev):
+    def _get_dev_uuids(self, dev):
         for adtype in self.SERVICE_CLASS_UUID_ADTYPES:
-            service_class_uuid = dev.getValueText(adtype)
-            if service_class_uuid:
-                logger.debug(self.SERVICE_CLASS_UUID_ADTYPES[adtype])
-                return UUID(service_class_uuid)
+            service_class_uuids = dev.getValue(adtype)
+            if service_class_uuids:
+                for u in service_class_uuids:
+                    a = self.SERVICE_CLASS_UUID_ADTYPES[adtype]
+                    logger.debug(f"service class uuid for {a}/{adtype}: {u}")
+                return service_class_uuids
         return None
 
     def matches(self, dev, filters):
         """
         Check if the found BLE device matches the filters Scratch specifies.
         """
-        logger.debug(f"in matches {dev.addr} {filters}")
+        logger.debug(f"in matches {dev.address} {filters}")
         for f in filters:
             if 'services' in f:
                 for s in f['services']:
                     logger.debug(f"service to check: {s}")
                     given_uuid = s
-                    logger.debug(f"given: {given_uuid}")
-                    dev_uuid = self._get_dev_uuid(dev)
-                    if not dev_uuid:
+                    logger.debug(f"given UUID: {given_uuid} hash={UUID(given_uuid).__hash__()}")
+                    dev_uuids = self._get_dev_uuids(dev)
+                    if not dev_uuids:
                         continue
-                    logger.debug(f"dev: {dev_uuid}")
-                    logger.debug(given_uuid == dev_uuid)
-                    if given_uuid == dev_uuid:
-                        logger.debug("match...")
-                        return True
+                    for u in dev_uuids:
+                        logger.debug(f"dev UUID: {u} hash={u.__hash__()}")
+                        logger.debug(given_uuid == u)
+                        if given_uuid == u:
+                            logger.debug("match...")
+                            return True
             if 'namePrefix' in f:
-                # 0x08: Shortened Local Name
-                deviceName = dev.getValueText(0x08)
-                if not deviceName:
-                    continue
-                logger.debug(f"Name of \"{deviceName}\" begins with: \"{f['namePrefix']}\"?")
-                if(deviceName.startswith(f['namePrefix'])):
-                    logger.debug("Yes")
-                    return True
-                logger.debug("No")
+                logger.debug(f"given namePrefix: {f['namePrefix']}")
+                deviceName = dev.getValueText(ScanEntry.SHORT_LOCAL_NAME)
+                if deviceName:
+                    logger.debug(f"SHORT_LOCAL_NAME: {deviceName}")
+                    if deviceName.startswith(f['namePrefix']):
+                        logger.debug(f"match...")
+                        return True
+                deviceName = dev.getValueText(ScanEntry.COMPLETE_LOCAL_NAME)
+                if deviceName:
+                    logger.debug(f"COMPLETE_LOCAL_NAME: {deviceName}")
+                    if deviceName.startswith(f['namePrefix']):
+                        logger.debug(f"match...")
+                        return True
             if 'name' in f or 'manufactureData' in f:
                 logger.error("name/manufactureData filters not implemented")
                 # TODO: implement other filters defined:
@@ -501,15 +351,23 @@ class BLESession(Session):
                 BLESession.found_devices.clear()
                 for i in range(self.MAX_SCANNER_IF):
                     scanner = Scanner(iface=i)
-                    try:
-                        devices = scanner.scan(10.0)
-                        for dev in devices:
-                            if self.matches(dev, params['filters']):
-                                BLESession.found_devices.append(dev)
-                                found = True
-                                logger.debug(f"BLE device found with iface #{i}");
-                    except BTLEManagementError as e:
-                        logger.debug(f"BLE iface #{i}: {e}");
+                    for j in range(scan_retry):
+                        try:
+                            logger.debug(f"start BLE scan: {self.scan_seconds} seconds")
+                            devices = scanner.scan(self.scan_seconds)
+                            for dev in devices:
+                                if self.matches(dev, params['filters']):
+                                    BLESession.found_devices.append(dev)
+                                    found = True
+                                    logger.debug(f"BLE device found with iface #{i}");
+                            if found:
+                                break
+                        except BTLEDisconnectError as de:
+                            logger.debug(f"BLE iface #{i}: {de}");
+                        except BTLEManagementError as me:
+                            logger.debug(f"BLE iface #{i}: {me}");
+                    if found:
+                        break
             else:
                 found = len(BLESession.found_devices) > 0
         return found
@@ -689,11 +547,13 @@ class BLESession(Session):
         return self.status == self.DONE
 
 async def ws_handler(websocket, path):
-    sessionTypes = { '/scratch/ble': BLESession, '/scratch/bt': BTSession }
+    global scan_seconds
+    sessionTypes = { '/scratch/bt': BTSession }
+    sessionTypes['/scratch/ble'] = ble.BLEDBusSession if dbus else BLESession
     try:
         logger.info(f"Start session for web socket path: {path}")
         loop = asyncio.get_event_loop()
-        session = sessionTypes[path](websocket, loop)
+        session = sessionTypes[path](websocket, loop, scan_seconds)
         await session.handle()
     except Exception as e:
         logger.error(f"Failure in session for web socket path: {path}")
@@ -715,18 +575,39 @@ def stack_trace():
          print(line)
 
 def main():
-    opts = [opt for opt in sys.argv[1:] if opt.startswith("-")]
-    if "-h" in opts:
-        print((f"Usage: {sys.argv[0]} [OPTS]\n"
-               "OPTS:\t-h Show this help.\n"
-               "\t-d Print debug messages."
-        ))
-        sys.exit(1)
-    elif "-d" in opts:
-        print("Print debug messages")
+    global scan_seconds
+    global scan_retry
+    global dbus
+    parser = argparse.ArgumentParser(description='start Scratch-link')
+    parser.add_argument('-d', '--debug', action='store_true',
+                        help='print debug messages')
+    parser.add_argument('-s', '--scan_seconds', type=float, default=10.0,
+                        help='specifiy duration to scan BLE devices in seconds')
+    parser.add_argument('-r', '--scan_retry', type=int, default=1,
+                        help='specifiy retry times to scan BLE devices')
+    parser.add_argument('-b', '--dbus', action='store_true',
+                        help='use DBus backend for BLE devices')
+    args = parser.parse_args()
+
+    logLevel = logging.INFO
+    if args.debug:
         logLevel = logging.DEBUG
-        handler.setLevel(logLevel)
-        logger.setLevel(logLevel)
+
+    formatter = logging.Formatter(fmt='%(asctime)s %(message)s')
+    handler = logging.StreamHandler()
+    handler.setLevel(logLevel)
+    handler.setFormatter(formatter)
+    logger.setLevel(logLevel)
+    logger.addHandler(handler)
+    logger.propagate = False
+
+    scan_seconds = args.scan_seconds
+    scan_retry = args.scan_retry
+    dbus = args.dbus
+    if args.debug:
+        logger.debug("Print debug messages")
+    logger.debug(f"set scan_seconds: {scan_seconds}")
+    logger.debug(f"set scan_retry: {scan_retry}")
 
     # Prepare certificate of the WSS server
     gencert.prep_cert()
@@ -738,7 +619,7 @@ def main():
     ssl_context.load_cert_chain(localhost_cer, localhost_key)
 
     start_server = websockets.serve(
-         ws_handler, "device-manager.scratch.mit.edu", 20110, ssl=ssl_context
+         ws_handler, HOSTNAME, 20110, ssl=ssl_context
     )
 
     while True:
@@ -749,7 +630,13 @@ def main():
         except KeyboardInterrupt as e:
             stack_trace()
             break
+        except socket.gaierror as e:
+            logger.error(f"{type(e).__name__}: {e}")
+            logger.info(f"Check internet connection to {HOSTNAME}. If not "
+                        f"available, add '127.0.0.1 {HOSTNAME}' to /etc/hosts.")
+            break
         except Exception as e:
+            logger.error(f"{type(e).__name__}: {e}")
             logger.info("Restarting scratch-link...")
 
 if __name__ == "__main__":

release.sh

@@ -4,13 +4,17 @@ usage() {
 	echo "Usage: ${0} COMMAND"
 	echo -e "COMMAND:"
 	echo -e "\tbuild"
+	echo -e "\tupload FILES"
 	echo -e "\tupload-testpypi FILES"
+	echo -e "examples:"
+	echo -e "\t${0} build"
+	echo -e "\t${0} upload dist/*0.2.6*"
 	exit 1
 }
 
 case ${1} in
 	build)
-		python setup.py sdist bdist_wheel
+		python -m build
 		;;
 	upload)
 		shift

setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setuptools.setup(
     name="pyscrlink",
-    version="0.2.2",
+    version="0.2.8",
     author="Shin'ichiro Kawasaki",
     author_email='kawasaki@juno.dti.ne.jp',
     description='Scratch-link for Linux with Python',
@@ -22,7 +22,6 @@ setuptools.setup(
     install_requires=[
         'websockets',
         'bluepy',
-        'pybluez',
         'pyOpenSSL',
         ],
     entry_points={