Currently in the esp32 port the size of the SPI flash must be configured at
build time, eg 4MiB, 8MiB, etc. Also, the esp32 partition table must be
configured at build time, which depends on the size of the SPI flash. A
bigger flash means more can be allocated to the user filesystem.
This commit makes it so the SPI flash size is automatically determined at
runtime, and the filesystem size is automatically set to take up as much
room as possible (a "vfs" partition is created automatically if it doesn't
exist).
This works by:
- Setting the SPI flash size to be 4MiB in the build (or some other value,
as long as the firmware app fits).
- Removing the vfs partition from the esp32 partition table (only nvs,
phy_init and firmware, and maybe romfs, remain in the partition table).
- At boot, query the physical size of the SPI flash and use that as the
actual size in the code.
- If it doesn't already exist, automatically create a "vfs" partition which
takes up the flash from the end of all existing partitions to the end of
flash.
This allows simplifying a lot of board configurations, and removing some
board variants that just change the flash size (to be done in a following
commit).
It's also fully backwards compatible, in the following sense:
- Existing boards with MicroPython firmware will continue to work with the
same filesystem, ie the filesystem won't be erased when the firmware is
updated.
- If a user has a custom esp32 partition table and installs MicroPython as
a bare app into the app partition, the new MicroPython firmware will
honour the esp32 partition table and use either "vfs" or "ffat"
partitions as the filesystem.
Signed-off-by: Damien George <damien@micropython.org>
Testing shows that for frequencies which the esp8266 can handle -- up to
about 1kHz -- `machine.time_pulse_us()` now gives more accurate results.
Prior to this commit it would measure on average about 1us lower, but now
the average is much closer to the true value. For example a pulse that is
1000us long, it would measure between 998 and 1000us. Now it measures
between 999us and 1001us.
Signed-off-by: Damien George <damien@micropython.org>
This implementation is based on the esp8266 custom implementation, and
further optimised for size and accuracy.
Testing on PYBD_SF2 and RPI_PICO2_W shows that it is at least as good as
the original implementation in performance.
Signed-off-by: Damien George <damien@micropython.org>
Reuse the `create_test_report()` function from `run-tests.py` to generate a
`_result.json` file summarising the test run.
Signed-off-by: Damien George <damien@micropython.org>
Reuse the `create_test_report()` function from `run-tests.py` to generate a
`_result.json` file summarising the test run.
Signed-off-by: Damien George <damien@micropython.org>
Reuse the `create_test_report()` function from `run-tests.py` to generate a
`_result.json` file summarising the test run. If there's more than one
permutation of the test run, only the last result is saved.
Signed-off-by: Damien George <damien@micropython.org>
This commit factors existing code in `run-tests.py` into a new helper
function `create_test_report()`. That function prints out a summary of the
test run (eg number of tests passed, number failed, number skipped) and
creates the corresponding `_results.json` file.
This is done so `create_test_report()` can be reused by the other test
runners.
The `test_count` counter is now gone, and instead the number of passed plus
number of failed tests is used as an equivalent count.
For consistency this commit makes a minor change to the printed output of
`run-tests.py`: instead of printing a shorthand name for tests that failed
or skipped, it now prints the full name. Eg what was previously printed as
`attrtuple2` is now printed as `basics/attrtuple2.py`. This makes the
output a little longer (when there are failed/skipped tests) but helps to
disambiguate the test name, eg which directory it's in.
Signed-off-by: Damien George <damien@micropython.org>
This test was factored out from `random_extra.py` back in commit
6572029dc0, and the skip logic copied from
that file. But the skip logic needs to test that the `random` and
`uniform` functions exist, not `randint`.
This commit fixes that skip logic.
Signed-off-by: Damien George <damien@micropython.org>
The windows-2019 runner has been deprecated by GitHub, so stop using that.
Also take the chance to stop using windows-2022 and just use windows-latest
everywhere.
Signed-off-by: Damien George <damien@micropython.org>
If a complex literal had a negative real part and a positive imaginary
part, it was not parsed properly because the imaginary part also came out
negative.
Includes a test of complex parsing, which fails without this fix.
Co-authored-by: ComplexSymbol <141301057+ComplexSymbol@users.noreply.github.com>
Signed-off-by: Jeff Epler <jepler@gmail.com>
As suggested by @dpgeorge, factor out part of array_construct to allow it
to be used for construction & extension.
Note that extending with a known-length list (or tuple) goes through the
slow path of calling array_extend once per element.
Fixes issue #7408.
Signed-off-by: Jeff Epler <jepler@gmail.com>
A few more bytes can be saved by not using nested `if`s (4 bytes for
`build-MICROBIT/py/parsenum.o`, 8 bytes for RPI_PICO firmware).
This commit is better viewed with whitespace changes hidden, because
two blocks were reindented (e.g., `git show -b`).
Signed-off-by: Jeff Epler <jepler@gmail.com>
By avoiding two different checks of the string length, code size is reduced
without changing behavior: Some invalid float/complex strings like "ix"
will get handled just like "xx" in the main number literal parsing code
instead.
The optimizer alone couldn't remove the reundant comparisons because it
couldn't make a transformation that let an invalid string like "ix" pass
into the generic number parsing code.
Signed-off-by: Jeff Epler <jepler@gmail.com>
If the error reporting mode is at least "normal", report a failure due to a
complex regex with a different message.
Fixes issue #17150.
Signed-off-by: Jeff Epler <jepler@gmail.com>
Add support for R_ARM_ABS32 relocations in native .mpy files. These can be
rewritten in the same way that data relocations are.
Fixes issue #14430.
Signed-off-by: Damien George <damien@micropython.org>
With this change, all tests (except thread tests) now pass on RPI_PICO when
using the native emitter:
(plug in RPI_PICO)
$ cd tests
$ ./run-tests.py -t a0 --via-mpy --emit native
Signed-off-by: Damien George <damien@micropython.org>
This commit lets the native emitter backend extends the range of the
BCC family of opcodes (BALL, BANY, BBC, BBS, BEQ, BGE, BGEU, BLT,
BLTU, BNALL, BNE, BNONE) from 8 bits to 18 bits.
The test suite contains some test files that, when compiled into native
code, would require BCC jumps outside the (signed) 8 bits range. In
this case either the MicroPython interpreter or mpy-cross would raise an
exception, not running the test when using the "--via-mpy --emit native"
command line options with the test runner.
This comes with a 3 bytes penalty on each forward jump, bringing the
footprint of those jumps to 6 bytes each, as a longer opcode sequence
has to be emitted to let jumps access a larger range. However, this is
slightly offset by the fact that backward jumps can be emitted with a
single opcode if the range is small enough (8-bits offset).
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the native emitter backend extends the range of the
BCCZ family of opcodes (BEQZ, BNEZ, BLTZ, BGEZ) from 12 bits to 18
bits.
The test suite contains some test files that, when compiled into native
code, would require BCCZ jumps outside the (signed) 12 bits range. In
this case either the MicroPython interpreter or mpy-cross would raise an
exception, not running the test when using the "--via-mpy --emit native"
command line options with the test runner.
This comes with a 3 bytes penalty on each forward jump, bringing the
footprint of those jumps to 6 bytes each, as a longer opcode sequence
has to be emitted to let jumps access a larger range. However, this is
slightly offset by the fact that backward jumps can be emitted with a
single opcode if the range is small enough (3 bytes for a 12-bits
offset).
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit fixes CI test runs for the `nanbox` target, which were
broken by the unconditional native emitter code output changes in the
test runner.
The `nanbox` configuration does not enable native emitters of any kind,
and with a full test run that includes executing emitted native code
things would break when doing CI runs.
This is worked around by introducing a common subset of tests that do
not involve the native emitter, and a more comprehensive set of tests
that include both non-emitter and emitter tests. The `nanbox` CI test
run will stop at the first subset, whilst other configurations will run
that and execute further tests.
Function names have been kept the same for steps that involve native
code, with the `nanbox` subset having another one. This should not
trigger any breakage in existing CI configurations or external scripts.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the test runner enumerate and run native tests if the
feature check fails but native tests were explicitly requested from the
command line.
The old behaviour would disable native tests anyway if the feature check
failed, however this hid a bug in the x86 native emitter that would be
triggered even during the feature check. That meant the test suite
would pass on x86 even with a broken emitter, as those tests would have
been skipped anyway.
Now, if the user asks for native code it will get native code out of the
runner no matter what.
Co-authored-by: Damien George <damien@micropython.org>
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit performs a small refactoring on the Arm native emitter, by
renaming all but one instance of ASM_ARM_REG_R8 into REG_TEMP.
ASM_ARM_REG_R8 is the temporary register used by the emitter when
operations cannot overwrite the value of a particular register and some
extra storage is needed.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit extends the range for int-indexed load/store opcode
generators, making them emit correct code sequences for offsets that
span more than 12 bits.
This is necessary due to those generator bits being also used in the
Viper emitter, where it's more probable to reference offsets that can
not be embedded in the LDR/STR opcodes.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit removes redundant RV32 implementations of certain
int-indexed code generation operations (32-bit load/store and 16-bit
load).
Those operations were already available as part of the native emitter
API but were not exposed to the Viper code generator. As part of the
introduction of more specialised load and store API calls to
int-indexed Viper load/store generator bits, the existing native emitter
implementations are reused, thus making the Viper implementations
redundant.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit updates the existing specialised implementations for
int-indexed 32-bit load and store operations, and adds a specialised
implementation for int-indexed 16-bit load.
The 32-bit operations relied on the fact that their applicability was
limited to a specific range, falling back on a generic implementation
otherwise. Introducing a single entry point for each int-indexed
load/store operation size would break that assumption. Now those two
operations contain fallback code to generate working code by themselves
instead of raising an exception.
The 16-bit operation instead simply did not have any range check, but it
was not exposed directly to the Viper emitter. When a 16-bit
int-indexed load entry point was introduced, the existing implementation
would fail when accessing memory outside its 0..255 halfwords range. A
specialised implementation is now present, performing fewer operations
than the existing Viper emitter equivalent.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the Thumb native code generator backend emit ARMv7-M
specific opcodes for indexed load/store operations if possible.
Now T3 opcode encodings are used if the generator backend is configured
to allow emitting ARMv7-M opcodes and if the (unsigned) scaled index
fits in 12 bits. Or, in other words, LDR{B,H}.W and STR{B,H}.W opcodes
are now emitted if possible.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit extends the generic ASM API by adding the rest of the
ASM_{LOAD,STORE}[size]_REG_REG_OFFSET macros whenever applicable.
The Viper int-indexed load/store code generator was changed to use those
API functions if they are available, falling back to backend-specific
implementations if possible and ultimately to a generic implementation.
Right now all backends except for x64 implement load16, load32, and
store32 operations (x64 only implements load16).
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
Most extmod network drivers were being defined on a per-port basis,
duplicating code and making enabling a driver on a new port harder.
This consolidates extmod driver declarations and removes the existing
per-port definitions of them.
This commit has been verified to be a no-op in terms of firmware change.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
`tools/boardgen.py` is used by the `make-pins.py` scripts in many ports to
generate the pin definitions for the machine module.
In #17391 it was found that this is currently generating the C structs for
board pin definitions with inconsistent ordering (across different build
runs), which makes it sometimes impossible to get a consistent binary file
even for no change in source files.
This commit fixes that by sorting the board pin names alphabetically.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
This commit takes the QEMU/Arm CI build and test step and splits it into
three separate steps (bigendian, sabrelite, thumb), to allow them to run
in parallel.
Currently the QEMU/Arm CI build step would take up to 16 minutes, often
being the last step blocking a full test run. With this commit, when
the steps run in parallel the time it takes to complete the QEMU/Arm
build and test procedure is cut in half - taking between 8 to 9 minutes
depending on the CI runner load.
The existing `ci_build_and_test_arm` function has been removed, in
favour of having three separate functions - one per configuration. They
are called `ci_build_and_test_arm_bigendian`,
`ci_build_and_test_arm_sabrelite`, and `ci_build_and_test_arm_thumb`.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit updates the ADC to use the new driver `esp_adc/adc_oneshot.h`.
There are several errata notes about not being able to change the bit-width
of the ADCs certain chips. The only chip that can switch resolution to a
lower one is the normal ESP32. ESP32 C2 and S3 are stuck at 12 bits, while
S2 is at 13 bits.
On the S2, you can change the resolution, but it has no effect on the
resolution, rather, it prevents attenuation from working at all!
The resolution is set to the maximum possible for each SoC, with the ESP32
being the only one not throwing errors when trying to set the bit-width to
9, 10, 11 or 12 bits using `ADC.width(bits)`.
Signed-off-by: Damian Nowacki (purewack) bobimaster15@gmail.com
If the interrupt is not freed but merely disabled, instead of reallocating
it every time the timer is enabled again we can instead just re-enable it.
That means we're no longer setting the handler every time, and we need to
ensure it does not change. Doing so by adding an additional wrapper
function does not only solve that problem, it also allows us to remove
some code duplication and simplify how machine_uart uses the timer.
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
esp_intr_free is not safe to call from the timer ISR because it requires
the current task (the one the ISR interrupted) to be pinned to the same
core as the interrupt was allocated on. Merely disabling the ISR however is
safe since that only requires that we're currently running on the same core
(which the ISR always is), regardless of the current task.
This was causing deadlocks in machine_uart when the ISR happened to
interrupt a task that was not pinned to a specific core.
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
Some tests are just too big for targets that don't have much heap memory,
eg `tests/extmod/vfs_rom.py`. Other tests are too large because the target
doesn't have enough IRAM for native code, eg esp8266 running
`tests/micropython/viper_args.py`.
Previously, such tests were explicitly skipped on targets known to have
little memory, eg esp8266. But this doesn't scale to multiple targets, nor
to more and more tests which are too large.
This commit addresses that by adding logic to the test runner so it can
automatically skip tests when they don't fit in the target's memory. It
does this by prepending a `print('START TEST')` to every test, and if a
`MemoryError` occurs before that line is printed then the test was too big.
This works for standard tests, tests that go via .mpy files, and tests that
run in native emitter mode via .mpy files.
For tests that are too big, it prints `lrge <test name>` on the output,
and at the end prints them on a separate line of skipped tests so they can
be distinguished. They are also distinguished in the `_result.json` file
as a skipped test with reason "too large".
Signed-off-by: Damien George <damien@micropython.org>
This commit fixes a small yet harmless issue that occurs when invoking
`ci_native_mpy_modules_build` on a persistent environment, as only X64
MPY files would be removed by the cleaning process.
Now the correct architecture is passed at all times when cleaning before
building a natmod for a particular architecture, forcing a full build of
all files to better simulate the CI environment (where there's no state
persisted between runs for this step).
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the CI pipeline build all natmods for the Xtensa
target, now that ROM symbols can be used in the linking process.
The restriction was put in place due to build failures on certain
natmods for Xtensa, as ROM symbols would not be used, causing undefined
symbol errors at build time.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit adds an optional configuration option for the ESP8266 port
that, if the board rebooted due to a crash, will print to stdout some
information about the error that triggered the issue.
It is not possible using regular SDK functions to intercept errors and
print information at that stage, and the only error response from the
board is to reboot itself. This is the next best thing, print some
error information just once at boot time after the crash - the least
invasive option given the situation we're in.
This is disabled by default, and can be enabled by enabling
MICROPY_HW_HARD_FAULT_DEBUG in the port configuration - obviously with a
small increase in the firmware code footprint.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit provides the appropriate external symbol addresses to let
the "btree" example natmod build for the Xtensa platform.
On the ESP8266, unsigned integer division code isn't provided as part of
libgcc.a, libm.a, or libc.a, but it is instead provided by the ROM.
Regular builds inject the appropriate symbol addresses as part of the
linking process (see eagle.rom.addr.v6.ld), but natmods need this
information brought in from somewhere else.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit provides the appropriate external symbol addresses to let
the "deflate" example natmod build for the Xtensa platform.
Unlike other natmods that require an external symbol list to build
without bringing in the whole runtime libraries set, this natmod is
referencing the `__modsi3` symbol which was removed from the ESP8266's
SDK but not present in ROM. The latter only has a `__umodsi3`
implementation that only operates on unsigned values, and thus unable to
handle this natmod. Thus, the extended library resolution process is
enabled for this natmod as a `__modsi3` implementation is made available
that way (still using ROM symbols whenever possible). This also means
that symbols that appear in both ROM and external libraries sort of
co-exist in the final MPY file, with ROM symbols being used by natmod
code but the implementation from the library still exists in the final
MPY file, unused.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit provides the appropriate external symbol addresses to let
the "framebuf" example natmod build for the Xtensa platform.
On the ESP8266, integer division code isn't provided as part of
libgcc.a, libm.a, or libc.a, but it is instead provided by the ROM.
Regular builds inject the appropriate symbol addresses as part of the
linking process (see eagle.rom.addr.v6.ld), but natmods need this
information brought in from somewhere else.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit provides the appropriate external symbol addresses to let
the "random" example natmod build for the Xtensa platform.
On the ESP8266, signed integer division code isn't provided as part of
libgcc.a, libm.a, or libc.a, but it is instead provided by the ROM.
Regular builds inject the appropriate symbol addresses as part of the
linking process (see eagle.rom.addr.v6.ld), but natmods need this
information brought in from somewhere else.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit introduces a mechanism to customise the code that is
injected to the board when performing a native module import.
A new argument, "-b"/"--begin", is added so regular Python code can be
inserted in the injected fragment between the module file creation and
the effective module import. This is needed for running natmod tests on
ESP8266 as that board does not have enough memory to fit certain modules
unless additional configuration is performed.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets mpy_ld.py resolve symbols not only from the object
files involved in the linking process, or from compiler-supplied static
libraries, but also from a list of symbols referenced by an absolute
address (usually provided by the system's ROM).
This is needed for ESP8266 targets as some C stdlib functions are
provided by the MCU's own ROM code to reduce the final code footprint,
and therefore those functions' implementation was removed from the
compiler's support libraries. This means that unless `LINK_RUNTIME` is
set (which lets tooling look at more libraries to resolve symbols) the
build process will fail as tooling is unaware of the ROM symbols'
existence. With this change, fixed-address symbols can be exposed to
the symbol resolution step when performing natmod linking.
If there are symbols coming in from a fixed-address symbols list and
internal code or external libraries, the fixed-address symbol address
will take precedence in all cases.
Although this is - in theory - also working for the whole range of ESP32
MCUs, testing is currently limited to Xtensa processors and the example
natmods' makefiles only make use of this commit's changes for the
ESP8266 target.
Natmod builds can set the MPY_EXTERN_SYM_FILE variable pointing to a
linkerscript file containing a series of symbols (weak or strong) at a
fixed address; these symbols will then be used by the MicroPython
linker when packaging the natmod. If a different natmod build method is
used (eg. custom CMake scripts), `tools/mpy_ld.py` can now accept a
command line parameter called `--externs` (or its short variant `-e`)
that contains the path of a linkerscript file with the fixed-address
symbols to use when performing the linking process.
The linkerscript file parser can handle a very limited subset of
binutils's linkerscript syntax, namely just block comments, strong
symbols, and weak symbols. Each symbol must be in its own line for the
parser to succeed, empty lines or comment blocks are skipped. For an
example of what this parser was meant to handle, you can look at
`ports/esp8266/boards/eagle.rom.addr.v6.ld` and follow its format.
The natmod developer documentation is also updated to reflect the new
command line argument accepted by `mpy_ld.py` and the use cases for the
changes introduced by this commit.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
Currently, mpy-cross will crash if called as:
mpy-cross -X emit=native foo.py
because it tries to use the native emitter with no target architecture set.
Fix that by checking that an architecture is set when `-X emit=native` or
`-X emit=viper` is used.
Signed-off-by: Damien George <damien@micropython.org>
