Tag version 0.2.4

Signed-off-by: Shin'ichiro Kawasaki <kawasaki@juno.dti.ne.jp>

README.md

@@ -1,51 +1,65 @@
 # pyscrlink
 
-Pyscrlink is [Scratch-link](https://github.com/LLK/scratch-link) for Linux.
+Pyscrlink is a [Scratch-link](https://github.com/LLK/scratch-link) for Linux.
 Scratch-link is a software module which connects
-[Scratch](https://scratch.mit.edu/) and Bluetooth devices such as
-[micro:bit](https://microbit.org/). However, as of October 2020, it works only
-on Windows and MacOS, and cannot connect Scratch and micro:bit on Linux
-operating systems.
+[Scratch](https://scratch.mit.edu/) to Bluetooth devices such as
+[micro:bit](https://microbit.org/). However, as of October 2020, it only works
+on Windows and MacOS, and Linux operating systems can not connect Scratch and
+micro:bit.
 
-Pyscrlink allows Linux OSes to connect Scratch and bluetooth devices. It uses
-Linux Bluetooth protocol stack [Bluez](http://www.bluez.org/) and its python
-interfaces [pybluez](https://github.com/pybluez/pybluez) to handle Bluetooth,
-and [bluepy](https://github.com/IanHarvey/bluepy) to handle Bluetooth Low
-Energy, or BLE, connections. It is confirmed that pyscrlink connects Scratch
-3.0 and a micro:bit, Lego Mindstorms EV3, Lego WeDo and Lego Boost.
+Pyscrlink allows you to connect Scratch and bluetooth devices with the Linux
+OSes. It uses the Linux Bluetooth protocol stack [Bluez](http://www.bluez.org/)
+and its python interfaces [pybluez](https://github.com/pybluez/pybluez) to
+handle Bluetooth, and [bluepy](https://github.com/IanHarvey/bluepy) to handle
+Bluetooth Low Energy (BLE) connections. It has been reported that pyscrlink
+connects Scratch 3.0 with micro:bit, LEGO Mindstorms EV3, LEGO WeDo, LEGO
+Boost, Intelino Smart Train and toio.
 
-Pyscrlink requires python version 3.6 and later to use websockets. If your
-system has python older than version 3.6, install newer version. If your Linux
-system has explicit command names python3 and pip3 for python version 3,
-use them in the instructions below.
+To use websockets, pyscrlink requires python version 3.6 or later. If your
+system has python older than version 3.6, install newer version. If your
+system has python 3 explicit command names python3 and pip3, use them in the
+steps below.
 
-Pyscrlink started in 2019 as "bluepy-scratch-link", which was a small work only
-for micro:bit and bluepy for BLE connections. After many contributions, it
-expanded coverage to other devices and pybluez for Bluetooth connections. The
-name "bluepy-scratch-link" was misleading that it indicate that it would depend
-only on bluepy. As of October 2020, the project was renamed from
-"bluepy-scratch-link" to "pyscrlink" to avoid the confusion.
+Pyscrlink was launched in 2019 as "bluepy-scratch-link". This was a small task
+dedicated to micro:bit and bluepy for BLE connection. After many contributions,
+it expanded coverage to pybluez with other devices for Bluetooth connectivity.
+It was misleading that the name "bluepy-scratch-link" indicates that it depends
+only on bluepy. As of October 2020, name of the project has been changed from
+"bluepy-scratch-link" to "pyscrlink" to avoid confusion.
 
 Confirmed Environments
 ----------------------
 The instructions below was confirmed with following devices and distros.
 Trial with other distros and feed-backs will be appreciated.
 
-The pyscrlink (former bluepy-scratch-link) was confirmed with following devices,
-Linux distros and browsers.
+Pyscrlink was confirmed with following devices, Linux distros and browsers.
 
 Devices:
-* micro:bit by @kawasaki
-* Lego Mindstorm EV3: by @chrisglencross
+* micro:bit
 
 Linux distros:
-* Arch Linux by @kawasaki
-* elementary OS 5.0 Juno by @kawasaki
-* Raspbian by @chirsglencross
+* Arch Linux
+* elementary OS 5.1 Hera
 
 Browsers:
-* FireFox by @kawasaki
-* Chromium by @kawasaki
+* Firefox
+* Chromium
+
+It was reported that pyscrlink (former bluepy-scratch-link) working with
+following devices and Linux distros.
+
+Devices:
+* LEGO Mindstorm EV3 by @chrisglencross
+* LEGO WeDo by @zhaowe, @KingBBQ
+* LEGO Boost and compatible devices by @laurentchar, @miguev, @jacquesdt, @n3storm
+* Intelino Smart Train by @ErrorJan
+* toio by @shimodash
+
+Linux distros:
+* Raspbian by @chrisglencross
+* Ubuntu 16.04 @jacquesdt
+* Ubuntu Studio 20.04 @miguev
+* Debian 11 @n3storm
 
 Installation
 ------------
@@ -61,7 +75,7 @@ Installation
 
     ```sh
     Ubuntu
-    $ sudo apt install bluez libbluetooth-dev libnss3-tools libcap2-bin
+    $ sudo apt install bluez libbluetooth-dev libnss3-tools libcap2-bin libglib2.0-dev
     Arch
     $ sudo pacman -S bluez bluez-utils nss libcap
     ```
@@ -74,7 +88,8 @@ Installation
     $ pip3 install pyscrlink
     ```
 
-4. For Bluetooth Low Energy (BLE) devices, set bluepy-helper capability.
+4. Set bluepy-helper capability.
+   This step is required for most of devices except LEGO Mindstorm EV3.
 
     ```
     $ bluepy_helper_cap
@@ -87,13 +102,13 @@ Installation
 5. For micro:bit, install Scratch-link hex on your device.
 
     * Download and unzip the [micro:bit Scratch Hex file](https://downloads.scratch.mit.edu/microbit/scratch-microbit-1.1.0.hex.zip).
-    * Flash the micro:bit over USB with the Scratch .Hex File, you will see the
+    * Flash the micro:bit over USB with the Scratch Hex File, you will see the
       five character name of the micro:bit scroll across the screen such as
       'zo9ev'.
 
 Usage
 -----
-1. For Lego Mindstorms EV3, pair your Linux PC to the EV3 brick.
+1. For LEGO Mindstorms EV3, pair your Linux PC to the EV3 brick.
 
    First, turn on the EV3 and ensure Bluetooth is enabled.
 
@@ -146,12 +161,15 @@ Usage
     ```sh
     $ scratch_link
     ```
+    If your device is toio, add "-s 1" option to the scratch_link command. It
+    allows the toio Do Visual Programming to connect to toio automatically.
 
-3. Connect scratch to micro:bit or Lego Mindstorms:
-    * Open FireFox or Chrome and access [Scratch 3.0](https://scratch.mit.edu/)
-    * Select the "Add Extension" button
-    * Select the extension for your device (e.g., micro:bit or Lego Mindstorms EV3 extension) and follow the prompts to connect
-    * Build your project with the extension blocks
+3. Connect scratch to the target device such as micro:bit or LEGO Mindstorms:
+    * Open FireFox or Chrome. (Make sure to run as the same user for scratch-link python script.)
+    * Access [Scratch 3.0](https://scratch.mit.edu/) and create your project.
+    * Select the "Add Extension" button.
+    * Select the extension for your device (e.g., micro:bit or Lego Mindstorms EV3 extension) and follow the prompts to connect.
+    * Build your project with the extension blocks.
 
 In Case You Fail to Connect
 ---------------------------
@@ -173,3 +191,44 @@ In Case You Fail to Connect
 
 3. If scratch_link.py says "failed to connect to BT device: [Errno 13] Permission denied",
    make sure to pair the bluetooth device to your PC before connecting to Scratch.
+
+4. To connect to multiple devices at the same time, make all the target devices
+   ready for scan at the first device scan. This is important for toio. The toio
+   allows a single project to connect to two toio devices.
+   * When the second device was prepared after the first device was connected, device scan can not find the second device.
+   * To scan and find the second device, disconnect connections for the first device beforehand.
+
+Issus Reporting
+---------------
+
+Please file issues to [GitHub issue tracker](https://github.com/kawasaki/pyscrlink/issues).
+
+Releases
+--------
+
+Release 0.2.4
+
+* Added -s option to specify BLE scan duration
+* Improved README.md
+
+Release 0.2.3
+
+* Fixed eternal loop caused by hostname resolve failure
+
+Release 0.2.2
+
+* Supported multiple device connections for toio
+* Improved session closure handling
+
+Release 0.2.1
+
+* Added libglib to required package list in README.md
+* Improved setcap and getcap tool finding
+
+Release 0.2.0
+
+* Latency issue fix for BLE devices' write characteristics
+
+Release 0.1.0
+
+* Initial release

pyscrlink/bluepy_helper_cap.py

@@ -22,10 +22,26 @@ logger.propagate = False
 # Check dependent tools
 DEPENDENT_TOOLS = {
     "setcap": "libcap2-bin (Ubuntu) or libcap (Arch)",
+    "getcap": "libcap2-bin (Ubuntu) or libcap (Arch)",
 }
 
+tools = {}
+
 for cmd in DEPENDENT_TOOLS:
-    if not shutil.which(cmd):
+    # find the tools in PATH
+    path = shutil.which(cmd)
+    if path:
+        tools[cmd] = path
+        logger.debug(f"{cmd} found: {path}")
+        continue
+    # find the tools out of PATH but in major directories
+    for d in ["/usr/bin", "/bin", "/usr/sbin", "/sbin"]:
+        path = d + '/' + cmd
+        if os.path.isfile(path) and os.access(path, os.X_OK):
+            tools[cmd] = path
+            logger.debug(f"{cmd} found: {path}")
+            break
+    if not cmd in tools:
         print(f"'{cmd}' not found. Install package {DEPENDENT_TOOLS[cmd]}.")
         sys.exit(1)
 
@@ -42,7 +58,7 @@ def helper_path():
 
 def is_set():
     path = helper_path()
-    p = subprocess.run(["getcap", path], stdout=subprocess.PIPE)
+    p = subprocess.run([tools["getcap"], path], stdout=subprocess.PIPE)
     if p.returncode != 0:
         logger.error(f"Failed to get capability of {path}")
         return False
@@ -53,8 +69,8 @@ def setcap():
     path = helper_path()
     if is_set():
         return True
-    p = subprocess.run(["sudo", "setcap", "cap_net_raw,cap_net_admin+eip",
-                        path])
+    p = subprocess.run(["sudo", tools["setcap"],
+                        "cap_net_raw,cap_net_admin+eip", path])
     if p.returncode !=0:
         logger.error(f"Failed to set capability to {path}")
         return False

pyscrlink/scratch_link.py

@@ -8,12 +8,14 @@ import asyncio
 import pathlib
 import ssl
 import websockets
+import socket
 import json
 import base64
 import logging
 import sys
 import signal
 import traceback
+import argparse
 
 # for Bluetooth (e.g. Lego EV3)
 import bluetooth
@@ -42,6 +44,9 @@ logger.setLevel(logLevel)
 logger.addHandler(handler)
 logger.propagate = False
 
+HOSTNAME="device-manager.scratch.mit.edu"
+scan_seconds=10.0
+
 class Session():
     """Base class for BTSession and BLESession"""
     def __init__(self, websocket, loop):
@@ -343,6 +348,11 @@ class BLESession(Session):
 
     MAX_SCANNER_IF = 3
 
+    found_devices = []
+    nr_connected = 0
+    scan_lock = threading.RLock()
+    scan_started = False
+
     class BLEThread(threading.Thread):
         """
         Separated thread to control notifications to Scratch.
@@ -358,7 +368,7 @@ class BLESession(Session):
                 logger.debug("loop in BLE thread")
                 if self.session.status == self.session.DISCOVERY:
                     logger.debug("send out found devices")
-                    devices = self.session.found_devices
+                    devices = BLESession.found_devices
                     for d in devices:
                         params = { 'rssi': d.rssi }
                         params['peripheralId'] = devices.index(d)
@@ -418,13 +428,20 @@ class BLESession(Session):
     def __init__(self, websocket, loop):
         super().__init__(websocket, loop)
         self.status = self.INITIAL
-        self.found_devices = []
         self.device = None
         self.deviceName = None
         self.perip = None
         self.delegate = None
+        self.characteristics_cache = []
 
     def close(self):
+        if self.status == self.CONNECTED:
+            BLESession.nr_connected -= 1
+            logger.info(f"BLE session disconnected")
+            logger.debug(f"BLE session connected={BLESession.nr_connected}")
+            if BLESession.nr_connected == 0:
+                logger.info("all BLE sessions disconnected")
+                BLESession.scan_started = False
         self.status = self.DONE
         if self.perip:
             logger.info("disconnect from the BLE peripheral: "
@@ -479,6 +496,31 @@ class BLESession(Session):
                 # ref: https://github.com/LLK/scratch-link/blob/develop/Documentation/BluetoothLE.md
         return False
 
+    def _scan_devices(self, params):
+        global scan_seconds
+        if BLESession.nr_connected > 0:
+            return len(BLESession.found_devices) > 0
+        found = False
+        with BLESession.scan_lock:
+            if not BLESession.scan_started:
+                BLESession.scan_started = True
+                BLESession.found_devices.clear()
+                for i in range(self.MAX_SCANNER_IF):
+                    scanner = Scanner(iface=i)
+                    try:
+                        logger.debug(f"start BLE scan: {scan_seconds} seconds")
+                        devices = scanner.scan(scan_seconds)
+                        for dev in devices:
+                            if self.matches(dev, params['filters']):
+                                BLESession.found_devices.append(dev)
+                                found = True
+                                logger.debug(f"BLE device found with iface #{i}");
+                    except BTLEManagementError as e:
+                        logger.debug(f"BLE iface #{i}: {e}");
+            else:
+                found = len(BLESession.found_devices) > 0
+        return found
+
     def _get_service(self, service_id):
         with self.lock:
             service = self.perip.getServiceByUUID(UUID(service_id))
@@ -490,6 +532,25 @@ class BLESession(Session):
             charas = self.perip.getCharacteristics(uuid=chara_id)
             return charas[0]
 
+    def _cache_characteristics(self):
+        if not self.perip:
+            return
+        with self.lock:
+            self.characteristics_cache = self.perip.getCharacteristics()
+        if not self.characteristics_cache:
+            logger.debug("Characteristics are not cached")
+
+    def _get_characteristic_cached(self, chara_id):
+        if not self.perip:
+            return None
+        if not self.characteristics_cache:
+            self._cache_characteristics()
+        if self.characteristics_cache:
+            for characteristic in self.characteristics_cache:
+                if characteristic.uuid == chara_id:
+                    return characteristic
+        return _get_characteristic(chara_id)
+
     def handle_request(self, method, params):
         """Handle requests from Scratch"""
         if self.delegate:
@@ -508,29 +569,21 @@ class BLESession(Session):
         if self.status == self.INITIAL and method == 'discover':
             if not bluepy_helper_cap.is_set():
                 logger.error("Capability is not set to bluepy helper.")
-                logger.error("Run bluepy_setcap.py with root privilege.")
+                logger.error("Run bluepy_helper_cap(.py).")
+                logger.error("e.g. $ bluepy_helper_cap")
                 logger.error("e.g. $ sudo bluepy_helper_cap.py")
                 sys.exit(1)
-            found_ifaces = 0
-            for i in range(self.MAX_SCANNER_IF):
-                scanner = Scanner(iface=i)
-                try:
-                    devices = scanner.scan(10.0)
-                    for dev in devices:
-                        if self.matches(dev, params['filters']):
-                            self.found_devices.append(dev)
-                    found_ifaces += 1
-                    logger.debug(f"BLE device found with iface #{i}");
-                except BTLEManagementError as e:
-                    logger.debug(f"BLE iface #{i}: {e}");
-
-            if found_ifaces == 0:
+            found = self._scan_devices(params)
+            if not found:
+                if BLESession.nr_connected > 0:
+                    err_msg = "Can not scan BLE devices. Disconnect other sessions."
+                elif len(BLESession.found_devices) == 0:
                     err_msg = "Can not scan BLE devices. Check BLE controller."
                 logger.error(err_msg);
                 res["error"] = { "message": err_msg }
                 self.status = self.DONE
 
-            if len(self.found_devices) == 0 and not err_msg:
+            if len(BLESession.found_devices) == 0 and not err_msg:
                 err_msg = (f"No BLE device found: {params['filters']}. "
                            "Check BLE device.")
                 res["error"] = { "message": err_msg }
@@ -544,11 +597,12 @@ class BLESession(Session):
 
         elif self.status == self.DISCOVERY and method == 'connect':
             logger.debug("connecting to the BLE device")
-            self.device = self.found_devices[params['peripheralId']]
+            self.device = BLESession.found_devices[params['peripheralId']]
             self.deviceName = self.device.getValueText(0x9) or self.device.getValueText(0x8)
             try:
                 self.perip = Peripheral(self.device)
                 logger.info(f"connected to the BLE peripheral: {self.deviceName}")
+                BLESession.found_devices.remove(self.device)
             except BTLEDisconnectError as e:
                 logger.error(f"failed to connect to the BLE device \"{self.deviceName}\": {e}")
                 self.status = self.DONE
@@ -556,8 +610,11 @@ class BLESession(Session):
             if self.perip:
                 res["result"] = None
                 self.status = self.CONNECTED
+                BLESession.nr_connected += 1
+                logger.debug(f"BLE session connected={BLESession.nr_connected}")
                 self.delegate = self.BLEDelegate(self)
                 self.perip.withDelegate(self.delegate)
+                self._cache_characteristics()
             else:
                 err_msg = f"BLE connect failed: {self.deviceName}"
                 res["error"] = { "message": err_msg }
@@ -595,7 +652,7 @@ class BLESession(Session):
             logger.debug("handle write request")
             service_id = params['serviceId']
             chara_id = params['characteristicId']
-            c = self._get_characteristic(chara_id)
+            c = self._get_characteristic_cached(chara_id)
             if not c or c.uuid != UUID(chara_id):
                 logger.error(f"Failed to get characteristic {chara_id}")
                 self.status = self.DONE
@@ -647,7 +704,8 @@ async def ws_handler(websocket, path):
         await session.handle()
     except Exception as e:
         logger.error(f"Failure in session for web socket path: {path}")
-        logger.error(e)
+        logger.error(f"{type(e).__name__}: {e}")
+        session.close()
 
 def stack_trace():
     print("in stack_trace")
@@ -664,18 +722,20 @@ def stack_trace():
          print(line)
 
 def main():
-    opts = [opt for opt in sys.argv[1:] if opt.startswith("-")]
-    if "-h" in opts:
-        print((f"Usage: {sys.argv[0]} [OPTS]\n"
-               "OPTS:\t-h Show this help.\n"
-               "\t-d Print debug messages."
-        ))
-        sys.exit(1)
-    elif "-d" in opts:
+    global scan_seconds
+    parser = argparse.ArgumentParser(description='start Scratch-link')
+    parser.add_argument('-d', '--debug', action='store_true',
+                        help='print debug messages')
+    parser.add_argument('-s', '--scan_seconds', type=float, default=10.0,
+                        help='specifiy duration to scan BLE devices in seconds')
+    args = parser.parse_args()
+    if args.debug:
         print("Print debug messages")
         logLevel = logging.DEBUG
         handler.setLevel(logLevel)
         logger.setLevel(logLevel)
+    scan_seconds = args.scan_seconds
+    logger.debug(f"set scan_seconds: {scan_seconds}")
 
     # Prepare certificate of the WSS server
     gencert.prep_cert()
@@ -687,7 +747,7 @@ def main():
     ssl_context.load_cert_chain(localhost_cer, localhost_key)
 
     start_server = websockets.serve(
-         ws_handler, "device-manager.scratch.mit.edu", 20110, ssl=ssl_context
+         ws_handler, HOSTNAME, 20110, ssl=ssl_context
     )
 
     while True:
@@ -698,7 +758,13 @@ def main():
         except KeyboardInterrupt as e:
             stack_trace()
             break
+        except socket.gaierror as e:
+            logger.error(f"{type(e).__name__}: {e}")
+            logger.info(f"Check internet connection to {HOSTNAME}. If not "
+                        f"available, add '127.0.0.1 {HOSTNAME}' to /etc/hosts.")
+            break
         except Exception as e:
+            logger.error(f"{type(e).__name__}: {e}")
             logger.info("Restarting scratch-link...")
 
 if __name__ == "__main__":

setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setuptools.setup(
     name="pyscrlink",
-    version="0.1b4",
+    version="0.2.4",
     author="Shin'ichiro Kawasaki",
     author_email='kawasaki@juno.dti.ne.jp',
     description='Scratch-link for Linux with Python',