README.md: Add description about -r option for scan failure

Add an item to "In Case You Fail to Connect" section to describe that
the "-r" option may help to address device scan failure.

Signed-off-by: Shin'ichiro Kawasaki <kawasaki@juno.dti.ne.jp>

.gitignore

@@ -1 +1,5 @@
-/__pycache__
+__pycache__
+build
+dist
+pyscrlink/__pycache__
+pyscrlink.egg-info

README.md

@@ -9,11 +9,16 @@ micro:bit.
 
 Pyscrlink allows you to connect Scratch and bluetooth devices with the Linux
 OSes. It uses the Linux Bluetooth protocol stack [Bluez](http://www.bluez.org/)
-and its python interfaces [pybluez](https://github.com/pybluez/pybluez) to
+and [bluepy](https://github.com/IanHarvey/bluepy) to handle Bluetooth Low Energy
-handle Bluetooth, and [bluepy](https://github.com/IanHarvey/bluepy) to handle
+(BLE) connections. It has been reported that pyscrlink connects Scratch 3.0 with
-Bluetooth Low Energy (BLE) connections. It has been reported that pyscrlink
+micro:bit, LEGO WeDo, LEGO Boost and toio.
-connects Scratch 3.0 with micro:bit, LEGO Mindstorms EV3, LEGO WeDo, LEGO
+
-Boost, Intelino Smart Train and toio.
+Until version v0.2.5, pyscrlink supported Bluetooth Classic protocol using
+[pybluez](https://github.com/pybluez/pybluez). Unfortunately, pybluez is not
+well maintained and caused technical troubles. Then Bluetooth Classic protocol
+support is dropped from pyscrlink. This means that LEGO Mindstorm EV3 can not
+be connected with pyscrlink. Bluetooth Classic support is the improvement
+opportunity of pyscrlink.
 
 To use websockets, pyscrlink requires python version 3.6 or later. If your
 system has python older than version 3.6, install newer version. If your
@@ -39,26 +44,24 @@ Devices:
 
 Linux distros:
 * Arch Linux
-* elementary OS 5.1 Hera
 
 Browsers:
-* FireFox
+* Firefox
 * Chromium
 
 It was reported that pyscrlink (former bluepy-scratch-link) working with
 following devices and Linux distros.
 
 Devices:
-* LEGO Mindstorm EV3 by @chrisglencross
 * LEGO WeDo by @zhaowe, @KingBBQ
-* LEGO Boost by @laurentchar, @miguev, @jacquesdt
+* LEGO Boost and compatible devices by @laurentchar, @miguev, @jacquesdt, @n3storm
-* Intelino Smart Train by @ErrorJan
 * toio by @shimodash
 
 Linux distros:
 * Raspbian by @chrisglencross
 * Ubuntu 16.04 @jacquesdt
 * Ubuntu Studio 20.04 @miguev
+* Debian 11 @n3storm
 
 Installation
 ------------
@@ -87,7 +90,7 @@ Installation
     $ pip3 install pyscrlink
     ```
 
-4. For Bluetooth Low Energy (BLE) devices, set bluepy-helper capability.
+4. Set bluepy-helper capability.
 
     ```
     $ bluepy_helper_cap
@@ -106,65 +109,18 @@ Installation
 
 Usage
 -----
-1. For LEGO Mindstorms EV3, pair your Linux PC to the EV3 brick.
+1. Start scratch-link python script.
-
-   First, turn on the EV3 and ensure Bluetooth is enabled.
-
-   Then, pair using your Linux desktop's the Bluetooth settings.
-
-   If using Gnome:
-      * Settings -> Bluetooth
-      * Click on the EV3 device name
-      * Accept the connection on EV3 brick
-      * Enter a matching PIN on EV3 brick and Linux PC. '1234' is the value Scratch suggests.
-      * Confirm EV3 status is "Disconnected" in Bluetooth settings
-
-   With a Raspberry Pi default Raspbian desktop, click the Bluetooth logo in the top right of the screen and
-   Add Device. Then follow the Gnome instructions. You will be warned that the Raspberry Pi
-   does not know how to talk to this device; that is not a problem.
-
-   Alternatively you can perform pairing from the command-line:
-   ```shell script
-   $ bluetoothctl
-
-   [bluetooth]# power on
-   Changing power on succeeded
-
-   [bluetooth]# pairable on
-   Changing pairable on succeeded
-
-   [bluetooth]# agent KeyboardOnly
-   Agent registered
-
-   [bluetooth]# devices
-   ...
-   Device 00:16:53:53:D3:19 EV3
-   ...
-
-   [bluetooth]# pair 00:16:53:53:D3:19
-   Attempting to pair with 00:16:53:53:D3:19
-
-   # Confirm pairing on the EV3 display, set PIN to 1234
-
-   Request PIN code
-   [agent] Enter PIN code: 1234
-   [CHG] Device 00:16:53:53:D3:19 Connected: yes
-   [CHG] Device 00:16:53:53:D3:19 Paired: yes
-   Pairing successful
-
-   [bluetooth]# quit
-   ```
-
-2. Start scratch-link python script.
     ```sh
     $ scratch_link
     ```
+    If your device is toio, add "-s 1" option to the scratch_link command. It
+    allows the toio Do Visual Programming to connect to toio automatically.
 
-3. Connect scratch to the target device such as micro:bit or LEGO Mindstorms:
+2. Connect scratch to the target device such as micro:bit:
     * Open FireFox or Chrome. (Make sure to run as the same user for scratch-link python script.)
     * Access [Scratch 3.0](https://scratch.mit.edu/) and create your project.
     * Select the "Add Extension" button.
-    * Select the extension for your device (e.g., micro:bit or Lego Mindstorms EV3 extension) and follow the prompts to connect.
+    * Select the extension for your device (e.g., micro:bit) and follow the prompts to connect.
     * Build your project with the extension blocks.
 
 In Case You Fail to Connect
@@ -185,10 +141,22 @@ In Case You Fail to Connect
     * If the service is not working, refer guide of your distro to set it up.
     * If the service is working, also check that /etc/bluetooth/main.conf sets AutoEnable=true.
 
-3. If scratch_link.py says "failed to connect to BT device: [Errno 13] Permission denied",
+3. If device scan still fails, use -r option to retry device scan.
+    The command line below does device scan twice. Each scan takes 10 seconds.
+    ```
+    $ scratch_link -r 2
+    ```
+    It would be good to use -s option together to reduce each scan duration.
+    The command line below does 3 seconds device scan twice.
+
+    ```
+    $ scratch_link -r 2 -s 3
+    ```
+
+4. If scratch_link.py says "failed to connect to BT device: [Errno 13] Permission denied",
    make sure to pair the bluetooth device to your PC before connecting to Scratch.
 
-4. To connect to multiple devices at the same time, make all the target devices
+5. To connect to multiple devices at the same time, make all the target devices
    ready for scan at the first device scan. This is important for toio. The toio
    allows a single project to connect to two toio devices.
    * When the second device was prepared after the first device was connected, device scan can not find the second device.
@@ -202,6 +170,23 @@ Please file issues to [GitHub issue tracker](https://github.com/kawasaki/pyscrli
 Releases
 --------
 
+Release 0.2.6
+
+* Removed Bluetooth Classic and LEGO Mindstorm EV3 support
+
+Release 0.2.5
+
+* Fixed handling of multiple UUIDs for LEGO Boost
+
+Release 0.2.4
+
+* Added -s option to specify BLE scan duration
+* Improved README.md
+
+Release 0.2.3
+
+* Fixed eternal loop caused by hostname resolve failure
+
 Release 0.2.2
 
 * Supported multiple device connections for toio

pyscrlink/gencert.py

@@ -152,36 +152,37 @@ def prep_nss_cert(dir, cert, nickname):
     remove_cert(dir, nickname)
     add_cert(dir, cert, nickname)
 
-def prep_cert():
+def prep_cert_for_app(cert, app, search_path):
-    # Generate certification and key
+    """
-    gen_cert(cert_file_path, key_file_path)
+    Find a NSS DB in the search_path for the app and prepare the cert in the DB.
-
+    """
-    # Add certificate to FireFox
     nssdb = None
-    firefox_nss_path = os.path.join(homedir, ".mozilla/firefox/")
+    for root, dirs, files in os.walk(os.path.join(homedir, search_path)):
-    for root, dirs, files in os.walk(firefox_nss_path):
         for name in files:
             if not re.match("key.*\.db", name):
                 continue
             nssdb = root
-            if prep_nss_cert(nssdb, cert_file_path, SCRATCH_CERT_NICKNAME):
+            if prep_nss_cert(nssdb, cert, SCRATCH_CERT_NICKNAME):
-                logger.error(f"Failed to add certificate to FireFox NSS DB: {nssdb}")
+                logger.error(f"Failed to add certificate to {app}: {nssdb}")
                 sys.exit(3)
             else:
-                logger.info(f"Certificate is ready in FireFox NSS DB: {nssdb}")
+                logger.info(f"Certificate is ready in {app} NSS DB: {nssdb}")
     if not nssdb:
-        logger.info("FireFox NSS DB not found. Do not add certificate.")
+        logger.debug(f"NSS DB for {app} not found. Do not add certificate.")
 
-    # Add certificate to Chrome
+
-    nssdb = os.path.join(homedir, ".pki/nssdb")
+def prep_cert():
-    if os.path.isdir(nssdb):
+    # Generate certification and key
-        if prep_nss_cert(nssdb, cert_file_path, SCRATCH_CERT_NICKNAME):
+    gen_cert(cert_file_path, key_file_path)
-            logger.error(f"Failed to add certificate to Chrome")
+
-            sys.exit(4)
+    nss_dbs = {
-        else:
+        "FireFox": ".mozilla/firefox/",
-            logger.info("Certificate is ready for Chrome")
+        "FireFox(Snap)": "snap/firefox/common/.mozilla/firefox/",
-    else:
+        "Chrome": ".pki",
-        logger.info("Chrome NSS DB not found. Do not add certificate.")
+        "Chromium(Snap)": "snap/chromium",
+        }
+
+    [ prep_cert_for_app(cert_file_path, k, nss_dbs[k]) for k in nss_dbs ]
 
 if __name__ == "__main__":
     prep_cert()

pyscrlink/scratch_link.py

@@ -8,15 +8,14 @@ import asyncio
 import pathlib
 import ssl
 import websockets
+import socket
 import json
 import base64
 import logging
 import sys
 import signal
 import traceback
-
+import argparse
-# for Bluetooth (e.g. Lego EV3)
-import bluetooth
 
 # for BLESession (e.g. BBC micro:bit)
 from bluepy.btle import Scanner, UUID, Peripheral, DefaultDelegate
@@ -42,6 +41,9 @@ logger.setLevel(logLevel)
 logger.addHandler(handler)
 logger.propagate = False
 
+HOSTNAME="device-manager.scratch.mit.edu"
+scan_seconds=10.0
+
 class Session():
     """Base class for BTSession and BLESession"""
     def __init__(self, websocket, loop):
@@ -133,195 +135,6 @@ class Session():
 class BTSession(Session):
     """Manage a session for Bluetooth device"""
 
-    INITIAL = 1
-    DISCOVERY = 2
-    DISCOVERY_COMPLETE = 3
-    CONNECTED = 4
-    DONE = 5
-
-    # Split this into discovery thread and communication thread
-    # discovery thread should auto-terminate
-
-    class BTThread(threading.Thread):
-        """
-        Separated thread to control notifications to Scratch.
-        It handles device discovery notification in DISCOVERY status
-        and notifications from bluetooth devices in CONNECTED status.
-        """
-
-        class BTDiscoverer(bluetooth.DeviceDiscoverer):
-
-            def __init__(self, major_class, minor_class):
-                super().__init__()
-                self.major_class = major_class
-                self.minor_class = minor_class
-                self.found_devices = {}
-                self.done = False
-
-            def pre_inquiry(self):
-                self.done = False
-
-            def device_discovered(self, address, device_class, rssi, name):
-                logger.debug(f"Found device {name} addr={address} class={device_class} rssi={rssi}")
-                major_class = (device_class & 0x1F00) >> 8
-                minor_class = (device_class & 0xFF) >> 2
-                if major_class == self.major_class and minor_class == self.minor_class:
-                    self.found_devices[address] = (name, device_class, rssi)
-
-            def inquiry_complete(self):
-                self.done = True
-
-        def __init__(self, session, major_device_class, minor_device_class):
-            threading.Thread.__init__(self)
-            self.session = session
-            self.major_device_class = major_device_class
-            self.minor_device_class = minor_device_class
-            self.cancel_discovery = False
-            self.ping_time = None
-
-        def discover(self):
-            discoverer = self.BTDiscoverer(self.major_device_class, self.minor_device_class)
-            discoverer.find_devices(lookup_names=True)
-            while self.session.status == self.session.DISCOVERY and not discoverer.done and not self.cancel_discovery:
-                readable = select.select([discoverer], [], [], 0.5)[0]
-                if discoverer in readable:
-                    discoverer.process_event()
-                    for addr, (device_name, device_class, rssi) in discoverer.found_devices.items():
-                        logger.debug(f"notifying discovered {addr}: {device_name}")
-                        params = {"rssi": rssi, 'peripheralId': addr, 'name': device_name.decode("utf-8")}
-                        self.session.notify('didDiscoverPeripheral', params)
-                    discoverer.found_devices.clear()
-
-            if not discoverer.done:
-                discoverer.cancel_inquiry()
-
-        def run(self):
-            while self.session.status != self.session.DONE:
-
-                logger.debug("loop in BT thread")
-                current_time = int(round(time.time()))
-
-                if self.session.status == self.session.DISCOVERY and not self.cancel_discovery:
-                    logger.debug("in discovery status:")
-                    try:
-                        self.discover()
-                        self.ping_time = current_time + 5
-                    finally:
-                        self.session.status = self.session.DISCOVERY_COMPLETE
-
-                elif self.session.status == self.session.CONNECTED:
-                    logger.debug("in connected status:")
-                    sock = self.session.sock
-                    try:
-                        ready = select.select([sock], [], [], 1)
-                        if ready[0]:
-                            header = sock.recv(2)
-                            [msg_len] = struct.unpack("<H", header)
-                            msg_data = sock.recv(msg_len)
-                            data = header + msg_data
-                            params = {'message': base64.standard_b64encode(data).decode('utf-8'), "encoding": "base64"}
-                            self.session.notify('didReceiveMessage', params)
-                            self.ping_time = current_time + 5
-
-                    except Exception as e:
-                            logger.error(e)
-                            self.session.close()
-                            break
-
-                    # To avoid repeated lock by this single thread,
-                    # yield CPU to other lock waiting threads.
-                    time.sleep(0)
-                else:
-                    # Nothing to do:
-                    time.sleep(1)
-
-                # Terminate if we have lost websocket connection to Scratch (e.g. browser closed)
-                if self.ping_time is None or self.ping_time <= current_time:
-                    try:
-                        self.session.notify('ping', {})
-                        self.ping_time = current_time + 5
-                    except Exception as e:
-                        logger.error(e)
-                        self.session.close()
-                        break
-
-    def __init__(self, websocket, loop):
-        super().__init__(websocket, loop)
-        self.status = self.INITIAL
-        self.sock = None
-        self.bt_thread = None
-
-    def close(self):
-        self.status = self.DONE
-        if self.sock:
-            logger.info(f"disconnect to BT socket: {self.sock}")
-            self.sock.close()
-            self.sock = None
-
-    def __del__(self):
-        self.close()
-
-    def handle_request(self, method, params):
-        """Handle requests from Scratch"""
-        logger.debug("handle request to BT device")
-        logger.debug(method)
-        if len(params) > 0:
-            logger.debug(params)
-
-        res = { "jsonrpc": "2.0" }
-
-        if self.status == self.INITIAL and method == 'discover':
-            logger.debug("Starting async discovery")
-            self.status = self.DISCOVERY
-            self.bt_thread = self.BTThread(self, params["majorDeviceClass"], params["minorDeviceClass"])
-            self.bt_thread.start()
-            res["result"] = None
-
-        elif self.status in [self.DISCOVERY, self.DISCOVERY_COMPLETE] and method == 'connect':
-
-            # Cancel discovery
-            while self.status == self.DISCOVERY:
-                logger.debug("Cancelling discovery")
-                self.bt_thread.cancel_discovery = True
-                time.sleep(1)
-
-            addr = params['peripheralId']
-            logger.debug(f"connecting to the BT device {addr}")
-            try:
-                self.sock = bluetooth.BluetoothSocket(bluetooth.RFCOMM)
-                self.sock.connect((addr, 1))
-                logger.info(f"connected to BT device: {addr}")
-            except bluetooth.BluetoothError as e:
-                logger.error(f"failed to connect to BT device: {e}", exc_info=e)
-                self.status = self.DONE
-                self.sock = None
-
-            if self.sock:
-                res["result"] = None
-                self.status = self.CONNECTED
-            else:
-                err_msg = f"BT connect failed: {addr}"
-                res["error"] = { "message": err_msg }
-                self.status = self.DONE
-
-        elif self.status == self.CONNECTED and method == 'send':
-            logger.debug("handle send request")
-            if params['encoding'] != 'base64':
-                logger.error("encoding other than base 64 is not "
-                                 "yet supported: ", params['encoding'])
-            msg_bstr = params['message'].encode('ascii')
-            data = base64.standard_b64decode(msg_bstr)
-            self.sock.send(data)
-            res['result'] = len(data)
-
-        logger.debug(res)
-        return res
-
-    def end_request(self):
-        logger.debug(f"end_request of BTSession {self}")
-        return self.status == self.DONE
-
-
 class BLESession(Session):
     """
     Manage a session for Bluetooth Low Energy device such as micro:bit
@@ -448,12 +261,14 @@ class BLESession(Session):
     def __del__(self):
         self.close()
 
-    def _get_dev_uuid(self, dev):
+    def _get_dev_uuids(self, dev):
         for adtype in self.SERVICE_CLASS_UUID_ADTYPES:
-            service_class_uuid = dev.getValueText(adtype)
+            service_class_uuids = dev.getValue(adtype)
-            if service_class_uuid:
+            if service_class_uuids:
-                logger.debug(self.SERVICE_CLASS_UUID_ADTYPES[adtype])
+                for u in service_class_uuids:
-                return UUID(service_class_uuid)
+                    a = self.SERVICE_CLASS_UUID_ADTYPES[adtype]
+                    logger.debug(f"service class uuid for {a}/{adtype}: {u}")
+                return service_class_uuids
         return None
 
     def matches(self, dev, filters):
@@ -466,13 +281,14 @@ class BLESession(Session):
                 for s in f['services']:
                     logger.debug(f"service to check: {s}")
                     given_uuid = s
-                    logger.debug(f"given: {given_uuid}")
+                    logger.debug(f"given UUID: {given_uuid} hash={UUID(given_uuid).__hash__()}")
-                    dev_uuid = self._get_dev_uuid(dev)
+                    dev_uuids = self._get_dev_uuids(dev)
-                    if not dev_uuid:
+                    if not dev_uuids:
                         continue
-                    logger.debug(f"dev: {dev_uuid}")
+                    for u in dev_uuids:
-                    logger.debug(given_uuid == dev_uuid)
+                        logger.debug(f"dev UUID: {u} hash={u.__hash__()}")
-                    if given_uuid == dev_uuid:
+                        logger.debug(given_uuid == u)
+                        if given_uuid == u:
                             logger.debug("match...")
                             return True
             if 'namePrefix' in f:
@@ -492,6 +308,7 @@ class BLESession(Session):
         return False
 
     def _scan_devices(self, params):
+        global scan_seconds
         if BLESession.nr_connected > 0:
             return len(BLESession.found_devices) > 0
         found = False
@@ -501,15 +318,23 @@ class BLESession(Session):
                 BLESession.found_devices.clear()
                 for i in range(self.MAX_SCANNER_IF):
                     scanner = Scanner(iface=i)
+                    for j in range(scan_retry):
                         try:
-                        devices = scanner.scan(10.0)
+                            logger.debug(f"start BLE scan: {scan_seconds} seconds")
+                            devices = scanner.scan(scan_seconds)
                             for dev in devices:
                                 if self.matches(dev, params['filters']):
                                     BLESession.found_devices.append(dev)
                                     found = True
                                     logger.debug(f"BLE device found with iface #{i}");
-                    except BTLEManagementError as e:
+                            if found:
-                        logger.debug(f"BLE iface #{i}: {e}");
+                                break
+                        except BTLEDisconnectError as de:
+                            logger.debug(f"BLE iface #{i}: {de}");
+                        except BTLEManagementError as me:
+                            logger.debug(f"BLE iface #{i}: {me}");
+                    if found:
+                        break
             else:
                 found = len(BLESession.found_devices) > 0
         return found
@@ -715,18 +540,25 @@ def stack_trace():
          print(line)
 
 def main():
-    opts = [opt for opt in sys.argv[1:] if opt.startswith("-")]
+    global scan_seconds
-    if "-h" in opts:
+    global scan_retry
-        print((f"Usage: {sys.argv[0]} [OPTS]\n"
+    parser = argparse.ArgumentParser(description='start Scratch-link')
-               "OPTS:\t-h Show this help.\n"
+    parser.add_argument('-d', '--debug', action='store_true',
-               "\t-d Print debug messages."
+                        help='print debug messages')
-        ))
+    parser.add_argument('-s', '--scan_seconds', type=float, default=10.0,
-        sys.exit(1)
+                        help='specifiy duration to scan BLE devices in seconds')
-    elif "-d" in opts:
+    parser.add_argument('-r', '--scan_retry', type=int, default=1,
+                        help='specifiy retry times to scan BLE devices')
+    args = parser.parse_args()
+    if args.debug:
         print("Print debug messages")
         logLevel = logging.DEBUG
         handler.setLevel(logLevel)
         logger.setLevel(logLevel)
+    scan_seconds = args.scan_seconds
+    scan_retry = args.scan_retry
+    logger.debug(f"set scan_seconds: {scan_seconds}")
+    logger.debug(f"set scan_retry: {scan_retry}")
 
     # Prepare certificate of the WSS server
     gencert.prep_cert()
@@ -738,7 +570,7 @@ def main():
     ssl_context.load_cert_chain(localhost_cer, localhost_key)
 
     start_server = websockets.serve(
-         ws_handler, "device-manager.scratch.mit.edu", 20110, ssl=ssl_context
+         ws_handler, HOSTNAME, 20110, ssl=ssl_context
     )
 
     while True:
@@ -749,7 +581,13 @@ def main():
         except KeyboardInterrupt as e:
             stack_trace()
             break
+        except socket.gaierror as e:
+            logger.error(f"{type(e).__name__}: {e}")
+            logger.info(f"Check internet connection to {HOSTNAME}. If not "
+                        f"available, add '127.0.0.1 {HOSTNAME}' to /etc/hosts.")
+            break
         except Exception as e:
+            logger.error(f"{type(e).__name__}: {e}")
             logger.info("Restarting scratch-link...")
 
 if __name__ == "__main__":

release.sh

@@ -4,13 +4,17 @@ usage() {
 	echo "Usage: ${0} COMMAND"
 	echo -e "COMMAND:"
 	echo -e "\tbuild"
+	echo -e "\tupload FILES"
 	echo -e "\tupload-testpypi FILES"
+	echo -e "examples:"
+	echo -e "\t${0} build"
+	echo -e "\t${0} upload dist/*0.2.6*"
 	exit 1
 }
 
 case ${1} in
 	build)
-		python setup.py sdist bdist_wheel
+		python -m build
 		;;
 	upload)
 		shift

setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setuptools.setup(
     name="pyscrlink",
-    version="0.2.2",
+    version="0.2.6",
     author="Shin'ichiro Kawasaki",
     author_email='kawasaki@juno.dti.ne.jp',
     description='Scratch-link for Linux with Python',
@@ -22,7 +22,6 @@ setuptools.setup(
     install_requires=[
         'websockets',
         'bluepy',
-        'pybluez',
         'pyOpenSSL',
         ],
     entry_points={