gencert.py: Support Snap Firefox and Chromium

Recently Ubuntu provides Firefox and Chromium in the form of Snap
package, and they place NSS DB at different path from non-Snap package.
However, current gencert.py implementation supports only the NSS DB
paths for non-Snap packages. This results in HTTPS communication failure
between the browsers and scratch_link.

Support the NSS DB paths for the Snap packages. Add a new function
prep_cert_for_app() which takes application name and its NSS DB search
path. Call this function for list of browsers, covering both non-Snap
and Snap packages.

Signed-off-by: Shin'ichiro Kawasaki <kawasaki@juno.dti.ne.jp>

.gitignore

@@ -1 +1,5 @@
-/__pycache__
+__pycache__
+build
+dist
+pyscrlink/__pycache__
+pyscrlink.egg-info

README.md

@@ -9,11 +9,16 @@ micro:bit.
 
 Pyscrlink allows you to connect Scratch and bluetooth devices with the Linux
 OSes. It uses the Linux Bluetooth protocol stack [Bluez](http://www.bluez.org/)
-and its python interfaces [pybluez](https://github.com/pybluez/pybluez) to
-handle Bluetooth, and [bluepy](https://github.com/IanHarvey/bluepy) to handle
-Bluetooth Low Energy (BLE) connections. It has been reported that pyscrlink
-connects Scratch 3.0 with micro:bit, LEGO Mindstorms EV3, LEGO WeDo, LEGO
-Boost, Intelino Smart Train and toio.
+and [bluepy](https://github.com/IanHarvey/bluepy) to handle Bluetooth Low Energy
+(BLE) connections. It has been reported that pyscrlink connects Scratch 3.0 with
+micro:bit, LEGO WeDo, LEGO Boost and toio.
+
+Until version v0.2.5, pyscrlink supported Bluetooth Classic protocol using
+[pybluez](https://github.com/pybluez/pybluez). Unfortunately, pybluez is not
+well maintained and caused technical troubles. Then Bluetooth Classic protocol
+support is dropped from pyscrlink. This means that LEGO Mindstorm EV3 can not
+be connected with pyscrlink. Bluetooth Classic support is the improvement
+opportunity of pyscrlink.
 
 To use websockets, pyscrlink requires python version 3.6 or later. If your
 system has python older than version 3.6, install newer version. If your
@@ -39,7 +44,6 @@ Devices:
 
 Linux distros:
 * Arch Linux
-* elementary OS 5.1 Hera
 
 Browsers:
 * Firefox
@@ -49,10 +53,8 @@ It was reported that pyscrlink (former bluepy-scratch-link) working with
 following devices and Linux distros.
 
 Devices:
-* LEGO Mindstorm EV3 by @chrisglencross
 * LEGO WeDo by @zhaowe, @KingBBQ
 * LEGO Boost and compatible devices by @laurentchar, @miguev, @jacquesdt, @n3storm
-* Intelino Smart Train by @ErrorJan
 * toio by @shimodash
 
 Linux distros:
@@ -89,7 +91,6 @@ Installation
     ```
 
 4. Set bluepy-helper capability.
-   This step is required for most of devices except LEGO Mindstorm EV3.
 
     ```
     $ bluepy_helper_cap
@@ -108,67 +109,18 @@ Installation
 
 Usage
 -----
-1. For LEGO Mindstorms EV3, pair your Linux PC to the EV3 brick.
-
-   First, turn on the EV3 and ensure Bluetooth is enabled.
-
-   Then, pair using your Linux desktop's the Bluetooth settings.
-
-   If using Gnome:
-      * Settings -> Bluetooth
-      * Click on the EV3 device name
-      * Accept the connection on EV3 brick
-      * Enter a matching PIN on EV3 brick and Linux PC. '1234' is the value Scratch suggests.
-      * Confirm EV3 status is "Disconnected" in Bluetooth settings
-
-   With a Raspberry Pi default Raspbian desktop, click the Bluetooth logo in the top right of the screen and
-   Add Device. Then follow the Gnome instructions. You will be warned that the Raspberry Pi
-   does not know how to talk to this device; that is not a problem.
-
-   Alternatively you can perform pairing from the command-line:
-   ```shell script
-   $ bluetoothctl
-
-   [bluetooth]# power on
-   Changing power on succeeded
-
-   [bluetooth]# pairable on
-   Changing pairable on succeeded
-
-   [bluetooth]# agent KeyboardOnly
-   Agent registered
-
-   [bluetooth]# devices
-   ...
-   Device 00:16:53:53:D3:19 EV3
-   ...
-
-   [bluetooth]# pair 00:16:53:53:D3:19
-   Attempting to pair with 00:16:53:53:D3:19
-
-   # Confirm pairing on the EV3 display, set PIN to 1234
-
-   Request PIN code
-   [agent] Enter PIN code: 1234
-   [CHG] Device 00:16:53:53:D3:19 Connected: yes
-   [CHG] Device 00:16:53:53:D3:19 Paired: yes
-   Pairing successful
-
-   [bluetooth]# quit
-   ```
-
-2. Start scratch-link python script.
+1. Start scratch-link python script.
     ```sh
     $ scratch_link
     ```
     If your device is toio, add "-s 1" option to the scratch_link command. It
     allows the toio Do Visual Programming to connect to toio automatically.
 
-3. Connect scratch to the target device such as micro:bit or LEGO Mindstorms:
+2. Connect scratch to the target device such as micro:bit:
     * Open FireFox or Chrome. (Make sure to run as the same user for scratch-link python script.)
     * Access [Scratch 3.0](https://scratch.mit.edu/) and create your project.
     * Select the "Add Extension" button.
-    * Select the extension for your device (e.g., micro:bit or Lego Mindstorms EV3 extension) and follow the prompts to connect.
+    * Select the extension for your device (e.g., micro:bit) and follow the prompts to connect.
     * Build your project with the extension blocks.
 
 In Case You Fail to Connect
@@ -206,6 +158,14 @@ Please file issues to [GitHub issue tracker](https://github.com/kawasaki/pyscrli
 Releases
 --------
 
+Release 0.2.6
+
+* Removed Bluetooth Classic and LEGO Mindstorm EV3 support
+
+Release 0.2.5
+
+* Fixed handling of multiple UUIDs for LEGO Boost
+
 Release 0.2.4
 
 * Added -s option to specify BLE scan duration

pyscrlink/gencert.py

@@ -152,36 +152,37 @@ def prep_nss_cert(dir, cert, nickname):
     remove_cert(dir, nickname)
     add_cert(dir, cert, nickname)
 
-def prep_cert():
-    # Generate certification and key
-    gen_cert(cert_file_path, key_file_path)
-
-    # Add certificate to FireFox
+def prep_cert_for_app(cert, app, search_path):
+    """
+    Find a NSS DB in the search_path for the app and prepare the cert in the DB.
+    """
     nssdb = None
-    firefox_nss_path = os.path.join(homedir, ".mozilla/firefox/")
-    for root, dirs, files in os.walk(firefox_nss_path):
+    for root, dirs, files in os.walk(os.path.join(homedir, search_path)):
         for name in files:
             if not re.match("key.*\.db", name):
                 continue
             nssdb = root
-            if prep_nss_cert(nssdb, cert_file_path, SCRATCH_CERT_NICKNAME):
-                logger.error(f"Failed to add certificate to FireFox NSS DB: {nssdb}")
+            if prep_nss_cert(nssdb, cert, SCRATCH_CERT_NICKNAME):
+                logger.error(f"Failed to add certificate to {app}: {nssdb}")
                 sys.exit(3)
             else:
-                logger.info(f"Certificate is ready in FireFox NSS DB: {nssdb}")
+                logger.info(f"Certificate is ready in {app} NSS DB: {nssdb}")
     if not nssdb:
-        logger.info("FireFox NSS DB not found. Do not add certificate.")
+        logger.debug(f"NSS DB for {app} not found. Do not add certificate.")
 
-    # Add certificate to Chrome
-    nssdb = os.path.join(homedir, ".pki/nssdb")
-    if os.path.isdir(nssdb):
-        if prep_nss_cert(nssdb, cert_file_path, SCRATCH_CERT_NICKNAME):
-            logger.error(f"Failed to add certificate to Chrome")
-            sys.exit(4)
-        else:
-            logger.info("Certificate is ready for Chrome")
-    else:
-        logger.info("Chrome NSS DB not found. Do not add certificate.")
+
+def prep_cert():
+    # Generate certification and key
+    gen_cert(cert_file_path, key_file_path)
+
+    nss_dbs = {
+        "FireFox": ".mozilla/firefox/",
+        "FireFox(Snap)": "snap/firefox/common/.mozilla/firefox/",
+        "Chrome": ".pki",
+        "Chromium(Snap)": "snap/chromium",
+        }
+
+    [ prep_cert_for_app(cert_file_path, k, nss_dbs[k]) for k in nss_dbs ]
 
 if __name__ == "__main__":
     prep_cert()

pyscrlink/scratch_link.py

@@ -17,9 +17,6 @@ import signal
 import traceback
 import argparse
 
-# for Bluetooth (e.g. Lego EV3)
-import bluetooth
-
 # for BLESession (e.g. BBC micro:bit)
 from bluepy.btle import Scanner, UUID, Peripheral, DefaultDelegate
 from bluepy.btle import BTLEDisconnectError, BTLEManagementError
@@ -138,195 +135,6 @@ class Session():
 class BTSession(Session):
     """Manage a session for Bluetooth device"""
 
-    INITIAL = 1
-    DISCOVERY = 2
-    DISCOVERY_COMPLETE = 3
-    CONNECTED = 4
-    DONE = 5
-
-    # Split this into discovery thread and communication thread
-    # discovery thread should auto-terminate
-
-    class BTThread(threading.Thread):
-        """
-        Separated thread to control notifications to Scratch.
-        It handles device discovery notification in DISCOVERY status
-        and notifications from bluetooth devices in CONNECTED status.
-        """
-
-        class BTDiscoverer(bluetooth.DeviceDiscoverer):
-
-            def __init__(self, major_class, minor_class):
-                super().__init__()
-                self.major_class = major_class
-                self.minor_class = minor_class
-                self.found_devices = {}
-                self.done = False
-
-            def pre_inquiry(self):
-                self.done = False
-
-            def device_discovered(self, address, device_class, rssi, name):
-                logger.debug(f"Found device {name} addr={address} class={device_class} rssi={rssi}")
-                major_class = (device_class & 0x1F00) >> 8
-                minor_class = (device_class & 0xFF) >> 2
-                if major_class == self.major_class and minor_class == self.minor_class:
-                    self.found_devices[address] = (name, device_class, rssi)
-
-            def inquiry_complete(self):
-                self.done = True
-
-        def __init__(self, session, major_device_class, minor_device_class):
-            threading.Thread.__init__(self)
-            self.session = session
-            self.major_device_class = major_device_class
-            self.minor_device_class = minor_device_class
-            self.cancel_discovery = False
-            self.ping_time = None
-
-        def discover(self):
-            discoverer = self.BTDiscoverer(self.major_device_class, self.minor_device_class)
-            discoverer.find_devices(lookup_names=True)
-            while self.session.status == self.session.DISCOVERY and not discoverer.done and not self.cancel_discovery:
-                readable = select.select([discoverer], [], [], 0.5)[0]
-                if discoverer in readable:
-                    discoverer.process_event()
-                    for addr, (device_name, device_class, rssi) in discoverer.found_devices.items():
-                        logger.debug(f"notifying discovered {addr}: {device_name}")
-                        params = {"rssi": rssi, 'peripheralId': addr, 'name': device_name.decode("utf-8")}
-                        self.session.notify('didDiscoverPeripheral', params)
-                    discoverer.found_devices.clear()
-
-            if not discoverer.done:
-                discoverer.cancel_inquiry()
-
-        def run(self):
-            while self.session.status != self.session.DONE:
-
-                logger.debug("loop in BT thread")
-                current_time = int(round(time.time()))
-
-                if self.session.status == self.session.DISCOVERY and not self.cancel_discovery:
-                    logger.debug("in discovery status:")
-                    try:
-                        self.discover()
-                        self.ping_time = current_time + 5
-                    finally:
-                        self.session.status = self.session.DISCOVERY_COMPLETE
-
-                elif self.session.status == self.session.CONNECTED:
-                    logger.debug("in connected status:")
-                    sock = self.session.sock
-                    try:
-                        ready = select.select([sock], [], [], 1)
-                        if ready[0]:
-                            header = sock.recv(2)
-                            [msg_len] = struct.unpack("<H", header)
-                            msg_data = sock.recv(msg_len)
-                            data = header + msg_data
-                            params = {'message': base64.standard_b64encode(data).decode('utf-8'), "encoding": "base64"}
-                            self.session.notify('didReceiveMessage', params)
-                            self.ping_time = current_time + 5
-
-                    except Exception as e:
-                            logger.error(e)
-                            self.session.close()
-                            break
-
-                    # To avoid repeated lock by this single thread,
-                    # yield CPU to other lock waiting threads.
-                    time.sleep(0)
-                else:
-                    # Nothing to do:
-                    time.sleep(1)
-
-                # Terminate if we have lost websocket connection to Scratch (e.g. browser closed)
-                if self.ping_time is None or self.ping_time <= current_time:
-                    try:
-                        self.session.notify('ping', {})
-                        self.ping_time = current_time + 5
-                    except Exception as e:
-                        logger.error(e)
-                        self.session.close()
-                        break
-
-    def __init__(self, websocket, loop):
-        super().__init__(websocket, loop)
-        self.status = self.INITIAL
-        self.sock = None
-        self.bt_thread = None
-
-    def close(self):
-        self.status = self.DONE
-        if self.sock:
-            logger.info(f"disconnect to BT socket: {self.sock}")
-            self.sock.close()
-            self.sock = None
-
-    def __del__(self):
-        self.close()
-
-    def handle_request(self, method, params):
-        """Handle requests from Scratch"""
-        logger.debug("handle request to BT device")
-        logger.debug(method)
-        if len(params) > 0:
-            logger.debug(params)
-
-        res = { "jsonrpc": "2.0" }
-
-        if self.status == self.INITIAL and method == 'discover':
-            logger.debug("Starting async discovery")
-            self.status = self.DISCOVERY
-            self.bt_thread = self.BTThread(self, params["majorDeviceClass"], params["minorDeviceClass"])
-            self.bt_thread.start()
-            res["result"] = None
-
-        elif self.status in [self.DISCOVERY, self.DISCOVERY_COMPLETE] and method == 'connect':
-
-            # Cancel discovery
-            while self.status == self.DISCOVERY:
-                logger.debug("Cancelling discovery")
-                self.bt_thread.cancel_discovery = True
-                time.sleep(1)
-
-            addr = params['peripheralId']
-            logger.debug(f"connecting to the BT device {addr}")
-            try:
-                self.sock = bluetooth.BluetoothSocket(bluetooth.RFCOMM)
-                self.sock.connect((addr, 1))
-                logger.info(f"connected to BT device: {addr}")
-            except bluetooth.BluetoothError as e:
-                logger.error(f"failed to connect to BT device: {e}", exc_info=e)
-                self.status = self.DONE
-                self.sock = None
-
-            if self.sock:
-                res["result"] = None
-                self.status = self.CONNECTED
-            else:
-                err_msg = f"BT connect failed: {addr}"
-                res["error"] = { "message": err_msg }
-                self.status = self.DONE
-
-        elif self.status == self.CONNECTED and method == 'send':
-            logger.debug("handle send request")
-            if params['encoding'] != 'base64':
-                logger.error("encoding other than base 64 is not "
-                                 "yet supported: ", params['encoding'])
-            msg_bstr = params['message'].encode('ascii')
-            data = base64.standard_b64decode(msg_bstr)
-            self.sock.send(data)
-            res['result'] = len(data)
-
-        logger.debug(res)
-        return res
-
-    def end_request(self):
-        logger.debug(f"end_request of BTSession {self}")
-        return self.status == self.DONE
-
-
 class BLESession(Session):
     """
     Manage a session for Bluetooth Low Energy device such as micro:bit
@@ -453,12 +261,14 @@ class BLESession(Session):
     def __del__(self):
         self.close()
 
-    def _get_dev_uuid(self, dev):
+    def _get_dev_uuids(self, dev):
         for adtype in self.SERVICE_CLASS_UUID_ADTYPES:
-            service_class_uuid = dev.getValueText(adtype)
-            if service_class_uuid:
-                logger.debug(self.SERVICE_CLASS_UUID_ADTYPES[adtype])
-                return UUID(service_class_uuid)
+            service_class_uuids = dev.getValue(adtype)
+            if service_class_uuids:
+                for u in service_class_uuids:
+                    a = self.SERVICE_CLASS_UUID_ADTYPES[adtype]
+                    logger.debug(f"service class uuid for {a}/{adtype}: {u}")
+                return service_class_uuids
         return None
 
     def matches(self, dev, filters):
@@ -471,15 +281,16 @@ class BLESession(Session):
                 for s in f['services']:
                     logger.debug(f"service to check: {s}")
                     given_uuid = s
-                    logger.debug(f"given: {given_uuid}")
-                    dev_uuid = self._get_dev_uuid(dev)
-                    if not dev_uuid:
+                    logger.debug(f"given UUID: {given_uuid} hash={UUID(given_uuid).__hash__()}")
+                    dev_uuids = self._get_dev_uuids(dev)
+                    if not dev_uuids:
                         continue
-                    logger.debug(f"dev: {dev_uuid}")
-                    logger.debug(given_uuid == dev_uuid)
-                    if given_uuid == dev_uuid:
-                        logger.debug("match...")
-                        return True
+                    for u in dev_uuids:
+                        logger.debug(f"dev UUID: {u} hash={u.__hash__()}")
+                        logger.debug(given_uuid == u)
+                        if given_uuid == u:
+                            logger.debug("match...")
+                            return True
             if 'namePrefix' in f:
                 # 0x08: Shortened Local Name
                 deviceName = dev.getValueText(0x08)

release.sh

@@ -4,13 +4,17 @@ usage() {
 	echo "Usage: ${0} COMMAND"
 	echo -e "COMMAND:"
 	echo -e "\tbuild"
+	echo -e "\tupload FILES"
 	echo -e "\tupload-testpypi FILES"
+	echo -e "examples:"
+	echo -e "\t${0} build"
+	echo -e "\t${0} upload dist/*0.2.6*"
 	exit 1
 }
 
 case ${1} in
 	build)
-		python setup.py sdist bdist_wheel
+		python -m build
 		;;
 	upload)
 		shift

setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setuptools.setup(
     name="pyscrlink",
-    version="0.2.4",
+    version="0.2.6",
     author="Shin'ichiro Kawasaki",
     author_email='kawasaki@juno.dti.ne.jp',
     description='Scratch-link for Linux with Python',
@@ -22,7 +22,6 @@ setuptools.setup(
     install_requires=[
         'websockets',
         'bluepy',
-        'pybluez',
         'pyOpenSSL',
         ],
     entry_points={