Tag version 0.2.5

Signed-off-by: Shin'ichiro Kawasaki <kawasaki@juno.dti.ne.jp>

README.md

@@ -1,51 +1,65 @@
 # pyscrlink
 
-Pyscrlink is [Scratch-link](https://github.com/LLK/scratch-link) for Linux.
+Pyscrlink is a [Scratch-link](https://github.com/LLK/scratch-link) for Linux.
 Scratch-link is a software module which connects
-[Scratch](https://scratch.mit.edu/) and Bluetooth devices such as
+[Scratch](https://scratch.mit.edu/) to Bluetooth devices such as
-[micro:bit](https://microbit.org/). However, as of October 2020, it works only
+[micro:bit](https://microbit.org/). However, as of October 2020, it only works
-on Windows and MacOS, and cannot connect Scratch and micro:bit on Linux
+on Windows and MacOS, and Linux operating systems can not connect Scratch and
-operating systems.
+micro:bit.
 
-Pyscrlink allows Linux OSes to connect Scratch and bluetooth devices. It uses
+Pyscrlink allows you to connect Scratch and bluetooth devices with the Linux
-Linux Bluetooth protocol stack [Bluez](http://www.bluez.org/) and its python
+OSes. It uses the Linux Bluetooth protocol stack [Bluez](http://www.bluez.org/)
-interfaces [pybluez](https://github.com/pybluez/pybluez) to handle Bluetooth,
+and its python interfaces [pybluez](https://github.com/pybluez/pybluez) to
-and [bluepy](https://github.com/IanHarvey/bluepy) to handle Bluetooth Low
+handle Bluetooth, and [bluepy](https://github.com/IanHarvey/bluepy) to handle
-Energy, or BLE, connections. It is confirmed that pyscrlink connects Scratch
+Bluetooth Low Energy (BLE) connections. It has been reported that pyscrlink
-3.0 and a micro:bit, Lego Mindstorms EV3, Lego WeDo and Lego Boost.
+connects Scratch 3.0 with micro:bit, LEGO Mindstorms EV3, LEGO WeDo, LEGO
+Boost, Intelino Smart Train and toio.
 
-Pyscrlink requires python version 3.6 and later to use websockets. If your
+To use websockets, pyscrlink requires python version 3.6 or later. If your
-system has python older than version 3.6, install newer version. If your Linux
+system has python older than version 3.6, install newer version. If your
-system has explicit command names python3 and pip3 for python version 3,
+system has python 3 explicit command names python3 and pip3, use them in the
-use them in the instructions below.
+steps below.
 
-Pyscrlink started in 2019 as "bluepy-scratch-link", which was a small work only
+Pyscrlink was launched in 2019 as "bluepy-scratch-link". This was a small task
-for micro:bit and bluepy for BLE connections. After many contributions, it
+dedicated to micro:bit and bluepy for BLE connection. After many contributions,
-expanded coverage to other devices and pybluez for Bluetooth connections. The
+it expanded coverage to pybluez with other devices for Bluetooth connectivity.
-name "bluepy-scratch-link" was misleading that it indicate that it would depend
+It was misleading that the name "bluepy-scratch-link" indicates that it depends
-only on bluepy. As of October 2020, the project was renamed from
+only on bluepy. As of October 2020, name of the project has been changed from
-"bluepy-scratch-link" to "pyscrlink" to avoid the confusion.
+"bluepy-scratch-link" to "pyscrlink" to avoid confusion.
 
 Confirmed Environments
 ----------------------
 The instructions below was confirmed with following devices and distros.
 Trial with other distros and feed-backs will be appreciated.
 
-The pyscrlink (former bluepy-scratch-link) was confirmed with following devices,
+Pyscrlink was confirmed with following devices, Linux distros and browsers.
-Linux distros and browsers.
 
 Devices:
-* micro:bit by @kawasaki
+* micro:bit
-* Lego Mindstorm EV3: by @chrisglencross
 
 Linux distros:
-* Arch Linux by @kawasaki
+* Arch Linux
-* elementary OS 5.0 Juno by @kawasaki
+* elementary OS 5.1 Hera
-* Raspbian by @chirsglencross
 
 Browsers:
-* FireFox by @kawasaki
+* Firefox
-* Chromium by @kawasaki
+* Chromium
+
+It was reported that pyscrlink (former bluepy-scratch-link) working with
+following devices and Linux distros.
+
+Devices:
+* LEGO Mindstorm EV3 by @chrisglencross
+* LEGO WeDo by @zhaowe, @KingBBQ
+* LEGO Boost and compatible devices by @laurentchar, @miguev, @jacquesdt, @n3storm
+* Intelino Smart Train by @ErrorJan
+* toio by @shimodash
+
+Linux distros:
+* Raspbian by @chrisglencross
+* Ubuntu 16.04 @jacquesdt
+* Ubuntu Studio 20.04 @miguev
+* Debian 11 @n3storm
 
 Installation
 ------------
@@ -61,7 +75,7 @@ Installation
 
     ```sh
     Ubuntu
-    $ sudo apt install bluez libbluetooth-dev libnss3-tools libcap2-bin
+    $ sudo apt install bluez libbluetooth-dev libnss3-tools libcap2-bin libglib2.0-dev
     Arch
     $ sudo pacman -S bluez bluez-utils nss libcap
     ```
@@ -74,7 +88,8 @@ Installation
     $ pip3 install pyscrlink
     ```
 
-4. For Bluetooth Low Energy (BLE) devices, set bluepy-helper capability.
+4. Set bluepy-helper capability.
+   This step is required for most of devices except LEGO Mindstorm EV3.
 
     ```
     $ bluepy_helper_cap
@@ -87,13 +102,13 @@ Installation
 5. For micro:bit, install Scratch-link hex on your device.
 
     * Download and unzip the [micro:bit Scratch Hex file](https://downloads.scratch.mit.edu/microbit/scratch-microbit-1.1.0.hex.zip).
-    * Flash the micro:bit over USB with the Scratch .Hex File, you will see the
+    * Flash the micro:bit over USB with the Scratch Hex File, you will see the
       five character name of the micro:bit scroll across the screen such as
       'zo9ev'.
 
 Usage
 -----
-1. For Lego Mindstorms EV3, pair your Linux PC to the EV3 brick.
+1. For LEGO Mindstorms EV3, pair your Linux PC to the EV3 brick.
 
    First, turn on the EV3 and ensure Bluetooth is enabled.
 
@@ -146,12 +161,15 @@ Usage
     ```sh
     $ scratch_link
     ```
+    If your device is toio, add "-s 1" option to the scratch_link command. It
+    allows the toio Do Visual Programming to connect to toio automatically.
 
-3. Connect scratch to micro:bit or Lego Mindstorms:
+3. Connect scratch to the target device such as micro:bit or LEGO Mindstorms:
-    * Open FireFox or Chrome and access [Scratch 3.0](https://scratch.mit.edu/)
+    * Open FireFox or Chrome. (Make sure to run as the same user for scratch-link python script.)
-    * Select the "Add Extension" button
+    * Access [Scratch 3.0](https://scratch.mit.edu/) and create your project.
-    * Select the extension for your device (e.g., micro:bit or Lego Mindstorms EV3 extension) and follow the prompts to connect
+    * Select the "Add Extension" button.
-    * Build your project with the extension blocks
+    * Select the extension for your device (e.g., micro:bit or Lego Mindstorms EV3 extension) and follow the prompts to connect.
+    * Build your project with the extension blocks.
 
 In Case You Fail to Connect
 ---------------------------
@@ -173,3 +191,48 @@ In Case You Fail to Connect
 
 3. If scratch_link.py says "failed to connect to BT device: [Errno 13] Permission denied",
    make sure to pair the bluetooth device to your PC before connecting to Scratch.
+
+4. To connect to multiple devices at the same time, make all the target devices
+   ready for scan at the first device scan. This is important for toio. The toio
+   allows a single project to connect to two toio devices.
+   * When the second device was prepared after the first device was connected, device scan can not find the second device.
+   * To scan and find the second device, disconnect connections for the first device beforehand.
+
+Issus Reporting
+---------------
+
+Please file issues to [GitHub issue tracker](https://github.com/kawasaki/pyscrlink/issues).
+
+Releases
+--------
+
+Release 0.2.5
+
+* Fixed handling of multiple UUIDs for LEGO Boost
+
+Release 0.2.4
+
+* Added -s option to specify BLE scan duration
+* Improved README.md
+
+Release 0.2.3
+
+* Fixed eternal loop caused by hostname resolve failure
+
+Release 0.2.2
+
+* Supported multiple device connections for toio
+* Improved session closure handling
+
+Release 0.2.1
+
+* Added libglib to required package list in README.md
+* Improved setcap and getcap tool finding
+
+Release 0.2.0
+
+* Latency issue fix for BLE devices' write characteristics
+
+Release 0.1.0
+
+* Initial release

pyscrlink/bluepy_helper_cap.py

@@ -22,10 +22,26 @@ logger.propagate = False
 # Check dependent tools
 DEPENDENT_TOOLS = {
     "setcap": "libcap2-bin (Ubuntu) or libcap (Arch)",
+    "getcap": "libcap2-bin (Ubuntu) or libcap (Arch)",
 }
 
+tools = {}
+
 for cmd in DEPENDENT_TOOLS:
-    if not shutil.which(cmd):
+    # find the tools in PATH
+    path = shutil.which(cmd)
+    if path:
+        tools[cmd] = path
+        logger.debug(f"{cmd} found: {path}")
+        continue
+    # find the tools out of PATH but in major directories
+    for d in ["/usr/bin", "/bin", "/usr/sbin", "/sbin"]:
+        path = d + '/' + cmd
+        if os.path.isfile(path) and os.access(path, os.X_OK):
+            tools[cmd] = path
+            logger.debug(f"{cmd} found: {path}")
+            break
+    if not cmd in tools:
         print(f"'{cmd}' not found. Install package {DEPENDENT_TOOLS[cmd]}.")
         sys.exit(1)
 
@@ -42,7 +58,7 @@ def helper_path():
 
 def is_set():
     path = helper_path()
-    p = subprocess.run(["getcap", path], stdout=subprocess.PIPE)
+    p = subprocess.run([tools["getcap"], path], stdout=subprocess.PIPE)
     if p.returncode != 0:
         logger.error(f"Failed to get capability of {path}")
         return False
@@ -53,8 +69,8 @@ def setcap():
     path = helper_path()
     if is_set():
         return True
-    p = subprocess.run(["sudo", "setcap", "cap_net_raw,cap_net_admin+eip",
+    p = subprocess.run(["sudo", tools["setcap"],
-                        path])
+                        "cap_net_raw,cap_net_admin+eip", path])
     if p.returncode !=0:
         logger.error(f"Failed to set capability to {path}")
         return False

pyscrlink/scratch_link.py

@@ -8,12 +8,14 @@ import asyncio
 import pathlib
 import ssl
 import websockets
+import socket
 import json
 import base64
 import logging
 import sys
 import signal
 import traceback
+import argparse
 
 # for Bluetooth (e.g. Lego EV3)
 import bluetooth
@@ -42,6 +44,9 @@ logger.setLevel(logLevel)
 logger.addHandler(handler)
 logger.propagate = False
 
+HOSTNAME="device-manager.scratch.mit.edu"
+scan_seconds=10.0
+
 class Session():
     """Base class for BTSession and BLESession"""
     def __init__(self, websocket, loop):
@@ -343,6 +348,11 @@ class BLESession(Session):
 
     MAX_SCANNER_IF = 3
 
+    found_devices = []
+    nr_connected = 0
+    scan_lock = threading.RLock()
+    scan_started = False
+
     class BLEThread(threading.Thread):
         """
         Separated thread to control notifications to Scratch.
@@ -358,7 +368,7 @@ class BLESession(Session):
                 logger.debug("loop in BLE thread")
                 if self.session.status == self.session.DISCOVERY:
                     logger.debug("send out found devices")
-                    devices = self.session.found_devices
+                    devices = BLESession.found_devices
                     for d in devices:
                         params = { 'rssi': d.rssi }
                         params['peripheralId'] = devices.index(d)
@@ -418,13 +428,20 @@ class BLESession(Session):
     def __init__(self, websocket, loop):
         super().__init__(websocket, loop)
         self.status = self.INITIAL
-        self.found_devices = []
         self.device = None
         self.deviceName = None
         self.perip = None
         self.delegate = None
+        self.characteristics_cache = []
 
     def close(self):
+        if self.status == self.CONNECTED:
+            BLESession.nr_connected -= 1
+            logger.info(f"BLE session disconnected")
+            logger.debug(f"BLE session connected={BLESession.nr_connected}")
+            if BLESession.nr_connected == 0:
+                logger.info("all BLE sessions disconnected")
+                BLESession.scan_started = False
         self.status = self.DONE
         if self.perip:
             logger.info("disconnect from the BLE peripheral: "
@@ -436,12 +453,14 @@ class BLESession(Session):
     def __del__(self):
         self.close()
 
-    def _get_dev_uuid(self, dev):
+    def _get_dev_uuids(self, dev):
         for adtype in self.SERVICE_CLASS_UUID_ADTYPES:
-            service_class_uuid = dev.getValueText(adtype)
+            service_class_uuids = dev.getValue(adtype)
-            if service_class_uuid:
+            if service_class_uuids:
-                logger.debug(self.SERVICE_CLASS_UUID_ADTYPES[adtype])
+                for u in service_class_uuids:
-                return UUID(service_class_uuid)
+                    a = self.SERVICE_CLASS_UUID_ADTYPES[adtype]
+                    logger.debug(f"service class uuid for {a}/{adtype}: {u}")
+                return service_class_uuids
         return None
 
     def matches(self, dev, filters):
@@ -454,13 +473,14 @@ class BLESession(Session):
                 for s in f['services']:
                     logger.debug(f"service to check: {s}")
                     given_uuid = s
-                    logger.debug(f"given: {given_uuid}")
+                    logger.debug(f"given UUID: {given_uuid} hash={UUID(given_uuid).__hash__()}")
-                    dev_uuid = self._get_dev_uuid(dev)
+                    dev_uuids = self._get_dev_uuids(dev)
-                    if not dev_uuid:
+                    if not dev_uuids:
                         continue
-                    logger.debug(f"dev: {dev_uuid}")
+                    for u in dev_uuids:
-                    logger.debug(given_uuid == dev_uuid)
+                        logger.debug(f"dev UUID: {u} hash={u.__hash__()}")
-                    if given_uuid == dev_uuid:
+                        logger.debug(given_uuid == u)
+                        if given_uuid == u:
                             logger.debug("match...")
                             return True
             if 'namePrefix' in f:
@@ -479,6 +499,31 @@ class BLESession(Session):
                 # ref: https://github.com/LLK/scratch-link/blob/develop/Documentation/BluetoothLE.md
         return False
 
+    def _scan_devices(self, params):
+        global scan_seconds
+        if BLESession.nr_connected > 0:
+            return len(BLESession.found_devices) > 0
+        found = False
+        with BLESession.scan_lock:
+            if not BLESession.scan_started:
+                BLESession.scan_started = True
+                BLESession.found_devices.clear()
+                for i in range(self.MAX_SCANNER_IF):
+                    scanner = Scanner(iface=i)
+                    try:
+                        logger.debug(f"start BLE scan: {scan_seconds} seconds")
+                        devices = scanner.scan(scan_seconds)
+                        for dev in devices:
+                            if self.matches(dev, params['filters']):
+                                BLESession.found_devices.append(dev)
+                                found = True
+                                logger.debug(f"BLE device found with iface #{i}");
+                    except BTLEManagementError as e:
+                        logger.debug(f"BLE iface #{i}: {e}");
+            else:
+                found = len(BLESession.found_devices) > 0
+        return found
+
     def _get_service(self, service_id):
         with self.lock:
             service = self.perip.getServiceByUUID(UUID(service_id))
@@ -490,6 +535,25 @@ class BLESession(Session):
             charas = self.perip.getCharacteristics(uuid=chara_id)
             return charas[0]
 
+    def _cache_characteristics(self):
+        if not self.perip:
+            return
+        with self.lock:
+            self.characteristics_cache = self.perip.getCharacteristics()
+        if not self.characteristics_cache:
+            logger.debug("Characteristics are not cached")
+
+    def _get_characteristic_cached(self, chara_id):
+        if not self.perip:
+            return None
+        if not self.characteristics_cache:
+            self._cache_characteristics()
+        if self.characteristics_cache:
+            for characteristic in self.characteristics_cache:
+                if characteristic.uuid == chara_id:
+                    return characteristic
+        return _get_characteristic(chara_id)
+
     def handle_request(self, method, params):
         """Handle requests from Scratch"""
         if self.delegate:
@@ -508,29 +572,21 @@ class BLESession(Session):
         if self.status == self.INITIAL and method == 'discover':
             if not bluepy_helper_cap.is_set():
                 logger.error("Capability is not set to bluepy helper.")
-                logger.error("Run bluepy_setcap.py with root privilege.")
+                logger.error("Run bluepy_helper_cap(.py).")
+                logger.error("e.g. $ bluepy_helper_cap")
                 logger.error("e.g. $ sudo bluepy_helper_cap.py")
                 sys.exit(1)
-            found_ifaces = 0
+            found = self._scan_devices(params)
-            for i in range(self.MAX_SCANNER_IF):
+            if not found:
-                scanner = Scanner(iface=i)
+                if BLESession.nr_connected > 0:
-                try:
+                    err_msg = "Can not scan BLE devices. Disconnect other sessions."
-                    devices = scanner.scan(10.0)
+                elif len(BLESession.found_devices) == 0:
-                    for dev in devices:
-                        if self.matches(dev, params['filters']):
-                            self.found_devices.append(dev)
-                    found_ifaces += 1
-                    logger.debug(f"BLE device found with iface #{i}");
-                except BTLEManagementError as e:
-                    logger.debug(f"BLE iface #{i}: {e}");
-
-            if found_ifaces == 0:
                     err_msg = "Can not scan BLE devices. Check BLE controller."
                 logger.error(err_msg);
                 res["error"] = { "message": err_msg }
                 self.status = self.DONE
 
-            if len(self.found_devices) == 0 and not err_msg:
+            if len(BLESession.found_devices) == 0 and not err_msg:
                 err_msg = (f"No BLE device found: {params['filters']}. "
                            "Check BLE device.")
                 res["error"] = { "message": err_msg }
@@ -544,11 +600,12 @@ class BLESession(Session):
 
         elif self.status == self.DISCOVERY and method == 'connect':
             logger.debug("connecting to the BLE device")
-            self.device = self.found_devices[params['peripheralId']]
+            self.device = BLESession.found_devices[params['peripheralId']]
             self.deviceName = self.device.getValueText(0x9) or self.device.getValueText(0x8)
             try:
                 self.perip = Peripheral(self.device)
                 logger.info(f"connected to the BLE peripheral: {self.deviceName}")
+                BLESession.found_devices.remove(self.device)
             except BTLEDisconnectError as e:
                 logger.error(f"failed to connect to the BLE device \"{self.deviceName}\": {e}")
                 self.status = self.DONE
@@ -556,8 +613,11 @@ class BLESession(Session):
             if self.perip:
                 res["result"] = None
                 self.status = self.CONNECTED
+                BLESession.nr_connected += 1
+                logger.debug(f"BLE session connected={BLESession.nr_connected}")
                 self.delegate = self.BLEDelegate(self)
                 self.perip.withDelegate(self.delegate)
+                self._cache_characteristics()
             else:
                 err_msg = f"BLE connect failed: {self.deviceName}"
                 res["error"] = { "message": err_msg }
@@ -595,7 +655,7 @@ class BLESession(Session):
             logger.debug("handle write request")
             service_id = params['serviceId']
             chara_id = params['characteristicId']
-            c = self._get_characteristic(chara_id)
+            c = self._get_characteristic_cached(chara_id)
             if not c or c.uuid != UUID(chara_id):
                 logger.error(f"Failed to get characteristic {chara_id}")
                 self.status = self.DONE
@@ -647,7 +707,8 @@ async def ws_handler(websocket, path):
         await session.handle()
     except Exception as e:
         logger.error(f"Failure in session for web socket path: {path}")
-        logger.error(e)
+        logger.error(f"{type(e).__name__}: {e}")
+        session.close()
 
 def stack_trace():
     print("in stack_trace")
@@ -664,18 +725,20 @@ def stack_trace():
          print(line)
 
 def main():
-    opts = [opt for opt in sys.argv[1:] if opt.startswith("-")]
+    global scan_seconds
-    if "-h" in opts:
+    parser = argparse.ArgumentParser(description='start Scratch-link')
-        print((f"Usage: {sys.argv[0]} [OPTS]\n"
+    parser.add_argument('-d', '--debug', action='store_true',
-               "OPTS:\t-h Show this help.\n"
+                        help='print debug messages')
-               "\t-d Print debug messages."
+    parser.add_argument('-s', '--scan_seconds', type=float, default=10.0,
-        ))
+                        help='specifiy duration to scan BLE devices in seconds')
-        sys.exit(1)
+    args = parser.parse_args()
-    elif "-d" in opts:
+    if args.debug:
         print("Print debug messages")
         logLevel = logging.DEBUG
         handler.setLevel(logLevel)
         logger.setLevel(logLevel)
+    scan_seconds = args.scan_seconds
+    logger.debug(f"set scan_seconds: {scan_seconds}")
 
     # Prepare certificate of the WSS server
     gencert.prep_cert()
@@ -687,7 +750,7 @@ def main():
     ssl_context.load_cert_chain(localhost_cer, localhost_key)
 
     start_server = websockets.serve(
-         ws_handler, "device-manager.scratch.mit.edu", 20110, ssl=ssl_context
+         ws_handler, HOSTNAME, 20110, ssl=ssl_context
     )
 
     while True:
@@ -698,7 +761,13 @@ def main():
         except KeyboardInterrupt as e:
             stack_trace()
             break
+        except socket.gaierror as e:
+            logger.error(f"{type(e).__name__}: {e}")
+            logger.info(f"Check internet connection to {HOSTNAME}. If not "
+                        f"available, add '127.0.0.1 {HOSTNAME}' to /etc/hosts.")
+            break
         except Exception as e:
+            logger.error(f"{type(e).__name__}: {e}")
             logger.info("Restarting scratch-link...")
 
 if __name__ == "__main__":

setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setuptools.setup(
     name="pyscrlink",
-    version="0.1b4",
+    version="0.2.5",
     author="Shin'ichiro Kawasaki",
     author_email='kawasaki@juno.dti.ne.jp',
     description='Scratch-link for Linux with Python',