`tools/boardgen.py` is used by the `make-pins.py` scripts in many ports to
generate the pin definitions for the machine module.
In #17391 it was found that this is currently generating the C structs for
board pin definitions with inconsistent ordering (across different build
runs), which makes it sometimes impossible to get a consistent binary file
even for no change in source files.
This commit fixes that by sorting the board pin names alphabetically.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
This commit takes the QEMU/Arm CI build and test step and splits it into
three separate steps (bigendian, sabrelite, thumb), to allow them to run
in parallel.
Currently the QEMU/Arm CI build step would take up to 16 minutes, often
being the last step blocking a full test run. With this commit, when
the steps run in parallel the time it takes to complete the QEMU/Arm
build and test procedure is cut in half - taking between 8 to 9 minutes
depending on the CI runner load.
The existing `ci_build_and_test_arm` function has been removed, in
favour of having three separate functions - one per configuration. They
are called `ci_build_and_test_arm_bigendian`,
`ci_build_and_test_arm_sabrelite`, and `ci_build_and_test_arm_thumb`.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit fixes a small yet harmless issue that occurs when invoking
`ci_native_mpy_modules_build` on a persistent environment, as only X64
MPY files would be removed by the cleaning process.
Now the correct architecture is passed at all times when cleaning before
building a natmod for a particular architecture, forcing a full build of
all files to better simulate the CI environment (where there's no state
persisted between runs for this step).
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the CI pipeline build all natmods for the Xtensa
target, now that ROM symbols can be used in the linking process.
The restriction was put in place due to build failures on certain
natmods for Xtensa, as ROM symbols would not be used, causing undefined
symbol errors at build time.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets mpy_ld.py resolve symbols not only from the object
files involved in the linking process, or from compiler-supplied static
libraries, but also from a list of symbols referenced by an absolute
address (usually provided by the system's ROM).
This is needed for ESP8266 targets as some C stdlib functions are
provided by the MCU's own ROM code to reduce the final code footprint,
and therefore those functions' implementation was removed from the
compiler's support libraries. This means that unless `LINK_RUNTIME` is
set (which lets tooling look at more libraries to resolve symbols) the
build process will fail as tooling is unaware of the ROM symbols'
existence. With this change, fixed-address symbols can be exposed to
the symbol resolution step when performing natmod linking.
If there are symbols coming in from a fixed-address symbols list and
internal code or external libraries, the fixed-address symbol address
will take precedence in all cases.
Although this is - in theory - also working for the whole range of ESP32
MCUs, testing is currently limited to Xtensa processors and the example
natmods' makefiles only make use of this commit's changes for the
ESP8266 target.
Natmod builds can set the MPY_EXTERN_SYM_FILE variable pointing to a
linkerscript file containing a series of symbols (weak or strong) at a
fixed address; these symbols will then be used by the MicroPython
linker when packaging the natmod. If a different natmod build method is
used (eg. custom CMake scripts), `tools/mpy_ld.py` can now accept a
command line parameter called `--externs` (or its short variant `-e`)
that contains the path of a linkerscript file with the fixed-address
symbols to use when performing the linking process.
The linkerscript file parser can handle a very limited subset of
binutils's linkerscript syntax, namely just block comments, strong
symbols, and weak symbols. Each symbol must be in its own line for the
parser to succeed, empty lines or comment blocks are skipped. For an
example of what this parser was meant to handle, you can look at
`ports/esp8266/boards/eagle.rom.addr.v6.ld` and follow its format.
The natmod developer documentation is also updated to reflect the new
command line argument accepted by `mpy_ld.py` and the use cases for the
changes introduced by this commit.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
If the USB serial device locks up, then writes to that device can hang
forever. That can make the test runner `tests/run-tests.py` lock up, among
other problems.
This commit introduces a 5 second write-timeout, and catches any OSError's
raised during `enter_raw_repl()`. Now, if a USB serial device locks up,
`enter_raw_repl()` will eventually raise an exception.
Signed-off-by: Damien George <damien@micropython.org>
This applies the mpremote commit 03fe9c55ea
to pyboard.py.
The `timeout_overall` is used in `enter_raw_repl()`. It prevents waiting
forever for a serial device that does not respond to the Ctrl-C/Ctrl-D/etc
commands and is constantly outputting data.
Signed-off-by: Damien George <damien@micropython.org>
This applies the mpremote commit 0d46e45a1f
to pyboard.py.
If the target does not return any data then `read_until()` will block
indefinitely. Fix this by making the initial read part of the general read
look, which always checks `inWaiting() > 0` before reading from the serial
device.
Also added the UART timeout to the constructor. This is not currently used
but may be used as an additional safeguard.
Signed-off-by: Damien George <damien@micropython.org>
Add `mpremote fs tree` command to show a tree of the device's files. It:
- Shows a treeview from current path or specified path.
- Uses the graph chars ("├── ", "└── ") (not configurable).
- Has the options:
-v/--verbose adds the serial device name to the top of the tree
-s/--size add a size to the files
-h/--human add a human readable size to the files
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
This version of the IDF uses about 1KB more IRAM and 1KB more DRAM on most
boards, but 6.5KB more DRAM usage on the S3. It seems that's due to a lot
of small increases in many components.
Signed-off-by: Ihor Nehrutsa <Ihor.Nehrutsa@gmail.com>
Previously, the navigation ended up messy when the (long) description of
the item became used as a 2nd level header, meaning that it was placed in
the navigation. Check for this when generating cpydiff so that new cases
don't sneak in unnoticed.
Signed-off-by: Jeff Epler <jepler@gmail.com>
The heading character for the difference title was always "~", but items
had been added which had just a single heading level. This made the
generated table of contents confused about heading levels, because heading
levels are not fixed in rst, but are inferred from the order they appear in
the document.
Signed-off-by: Jeff Epler <jepler@gmail.com>
In the syntax_space cpydiff, all the warnings were shown after the other
output. This is because the output always showed all of stdout first and
all of stdout second.
By running Python in unbuffered mode and using `stderr=STDOUT`, the two
streams are interleaved in exactly the order they're printed, so the
SyntaxWarnings are interleaved with the other output.
By using the `encoding=` argument of Popen, the need to explicitly convert
to utf-8 is avoided. The encoding of the input also becomes utf-8 in this
case, which all the test cases are (well, they're all ASCII, I think). As
in `run-tests.py`, setting PYTHONIOENCODING ensures the Python
interpreter's input and output are in utf-8, which is not always the case,
especially on Windows systems.
I spot-checked the generated doc pages and they all seemed to make sense
still.
Signed-off-by: Jeff Epler <jepler@gmail.com>
There is a bit of ambiguity as to how the prefix of the git subject line
should look like. Eg `py/vm: ...` vs `py/vm.c: ...` (whether the extension
should be there or not).
This commit makes the existing CI check of the git commit message stricter,
by applying extra rules to the prefix, the bit before the : in the subject
line. It now checks that the subject prefix:
- doesn't start with unwanted bits: ., /, ports/
- doesn't have an extension: .c, .h, .cpp, .js, .rst or .md
Full error messages are given when a rule does not pass.
This helps to reduce maintainer burden by applying stricter rules, to keep
the git commit history consistent.
Signed-off-by: Damien George <damien@micropython.org>
When using `mip install`, if a file that needs to be downloaded already
exists locally, then the hash of that local file will be computed and if it
matches the known hash of the remote file it will not be downloaded.
Hashes in mip are guaranteed unique, so this change should never leave
stale files on the filesystem.
This behaviour follows that of the `mip` package in `micropython-lib`.
Signed-off-by: Damien George <damien@micropython.org>
Updates the Zephyr port build instructions. The CI is updated to use
Zephyr docker image 0.27.4, SDK 0.17.0 and the latest Zephyr release
tag.
Tested on max32690fthr and frdm_k64f.
Signed-off-by: Maureen Helm <maureen.helm@analog.com>
Signed-off-by: Detlev Zundel <dzu@member.fsf.org>
Removes the risk of inadvertently deleting files on the host by preventing
the deletion of files via `rm -r` on the `/remote` vfs mount point.
Fixes issue #17147.
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
This rewrites the code that previously manually emitted and caught various
OSError subclasses with equivalent code that uses the errno name dictionary
to do this generically, and updates the exception handler in do_filesystem
to catch them in a similarly-generic fashion using os.strerror to retrieve
an appropriate error message text equivalent to the current messages.
Note that in the CPython environments where mpremote runs, the call to the
OSError constructor already returns an instance of the corresponding mapped
exception subtype.
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Because the `wbits` parameter was only added to `zlib.compress` in
CPython 3.11. Using `zlib.compressobj` makes the code compatible with much
older CPython versions.
Fixes issue #17140.
Signed-off-by: Damien George <damien@micropython.org>
Brings it into sync with a matching change to micropython-lib (which was
much older). Includes one small automatic fix.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Currently the tool allows writing an invalid ROMFS image, with a bad header
or images smaller than minimum size, and only checks the image extension.
This commit allows deploying a ROMFS with either a ".img" or ".romfs"
extension (in the future support may be added for other extensions that
have different semantics, eg a manifest), and validates the image header
before writing.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
Previously this information was recorded in a "status" field of the result,
but nothing ever parsed this result which led to non-differences not being
removed.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
If picotool is not installed, it's fetched and built when compiling each
rp2 board. And the "develop" branch of picotool is used instead of a
release. Installing it manually using the "master" branch means the latest
released version is used (instead of a possibly unstable development
version), and also makes building each rp2 board a little faster.
Signed-off-by: Damien George <damien@micropython.org>
This is a known limitation, so better to give a clear warning than a
catch-all AssertionError. Happens for example when trying to use
soft-float on ARCH=armv6m
Also give more details on the assertion for unknown relocations, such that
one can see which symbol it affects etc, to aid in debugging.
References issue #14430.
Signed-off-by: Jon Nordby <jononor@gmail.com>
If a ROMFS is mounted then "/rom/lib" is usually in `sys.path` before the
writable filesystem's "lib" entry. The ROMFS directory cannot be installed
to, so skip it if found.
Signed-off-by: Damien George <damien@micropython.org>
This commit removes the assumption made by the CI scripts that the
system-provided python executable is simply named "python". The scripts
will now look for a binary called "python3" first, and then fall back to
"python" if that is not found.
Whilst this is currently the case for the CI environment, there are no
guarantees for this going forward. For example minimal CI environments
set up by some developers, using the same base OS, have their python
executable called "python3".
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit expands the CI tests by checking whether the example native
modules are able to be built for the Xtensa architecture.
This was made possible by the changes to mpy_ld that allow symbol
resolution across standard compiler-provided libraries.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit introduces an additional symbol resolution mechanism to the
natmod linking process. This allows the build scripts to look for required
symbols into selected libraries that are provided by the compiler
installation (libgcc and libm at the moment).
For example, using soft-float code in natmods, whilst technically possible,
was not an easy process and required some additional work to pull it off.
With this addition all the manual (and error-prone) operations have been
automated and folded into `tools/mpy_ld.py`.
Both newlib and picolibc toolchains are supported, albeit the latter may
require a bit of extra configuration depending on the environment the build
process runs on. Picolibc's soft-float functions aren't in libm - in fact
the shipped libm is nothing but a stub - but they are inside libc. This is
usually not a problem as these changes cater for that configuration quirk,
but on certain compilers the include paths used to find libraries in may
not be updated to take Picolibc's library directory into account. The bare
metal RISC-V compiler shipped with the CI OS image (GCC 10.2.0 on Ubuntu
22.04LTS) happens to exhibit this very problem.
To work around that for CI builds, the Picolibc libraries' path is
hardcoded in the Makefile directives used by the linker, but this can be
changed by setting the PICOLIBC_ROOT environment library when building
natmods.
Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com>
Co-authored-by: Alessandro Gatti <a.gatti@frob.it>
These commands use the `vfs.rom_ioctl()` function to manage the ROM
partitions on a device, and create and deploy ROMFS images.
Signed-off-by: Damien George <damien@micropython.org>
This allows running mpy-tool using MicroPython itself.
An appropriate test is added to CI to make sure it continues to work.
Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com>
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This significantly speeds up readline on files opened directly from an
mpremote mount.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Changing runner OS can change Python version, and ESP-IDF installs are
keyed on ESP-IDF and Python version together.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
URLs in `package.json` may now be specified relative to the base URL of the
`package.json` file.
Relative URLs wil work for `package.json` files installed from the web as
well as local file paths.
Docs: update `docs/reference/packages.rst` to add documentation for:
- Installing packages from local filesystems (PR #12476); and
- Using relative URLs in the `package.json` file (PR #12477);
- Update the packaging example to encourage relative URLs as the default
in `package.json`.
Add `tools/mpremote/tests/test_mip_local_install.sh` to test the
installation of a package from local files using relative URLs in the
`package.json`.
Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
