This adds a compile-time flag to disable some "unsafe" and non-standard
typecodes in struct, array and related modules.
This is useful to turn off when fuzzing, as improper use of these typecodes
can crash MicroPython.
Signed-off-by: Jeff Epler <jepler@gmail.com>
Add a new MICROPY_COMP_CONST_FLOAT feature, enabled by in mpy-cross and
when compiling with MICROPY_CONFIG_ROM_LEVEL_CORE_FEATURES. The new
feature leverages the code of MICROPY_COMP_CONST_FOLDING to support folding
of floating point constants.
If MICROPY_COMP_MODULE_CONST is defined as well, math module constants are
made available at compile time. For example:
_DEG_TO_GRADIANT = const(math.pi / 180)
_INVALID_VALUE = const(math.nan)
A few corner cases had to be handled:
- The float const folding code should not fold expressions resulting into
complex results, as the mpy parser for complex immediates has
limitations.
- The constant generation code must distinguish between -0.0 and 0.0, which
are different even if C consider them as ==.
This change removes previous limitations on the use of `const()`
expressions that would result in floating point number, so the test cases
of micropython/const_error have to be updated.
Additional test cases have been added to cover the new repr() code (from a
previous commit). A few other simple test cases have been added to handle
the use of floats in `const()` expressions, but the float folding code
itself is also tested when running general float test cases, as float
expressions often get resolved at compile-time (with this change).
Signed-off-by: Yoctopuce dev <dev@yoctopuce.com>
Following discussions in PR #16666, this commit updates the float
formatting code to improve the `repr` reversibility, i.e. the percentage of
valid floating point numbers that do parse back to the same number when
formatted by `repr` (in CPython it's 100%).
This new code offers a choice of 3 float conversion methods, depending on
the desired tradeoff between code size and conversion precision:
- BASIC method is the smallest code footprint
- APPROX method uses an iterative method to approximate the exact
representation, which is a bit slower but but does not have a big impact
on code size. It provides `repr` reversibility on >99.8% of the cases in
double precision, and on >98.5% in single precision (except with REPR_C,
where reversibility is 100% as the last two bits are not taken into
account).
- EXACT method uses higher-precision floats during conversion, which
provides perfect results but has a higher impact on code size. It is
faster than APPROX method, and faster than the CPython equivalent
implementation. It is however not available on all compilers when using
FLOAT_IMPL_DOUBLE.
Here is the table comparing the impact of the three conversion methods on
code footprint on PYBV10 (using single-precision floats) and reversibility
rate for both single-precision and double-precision floats. The table
includes current situation as a baseline for the comparison:
PYBV10 REPR_C FLOAT DOUBLE
current = 364688 12.9% 27.6% 37.9%
basic = 364812 85.6% 60.5% 85.7%
approx = 365080 100.0% 98.5% 99.8%
exact = 366408 100.0% 100.0% 100.0%
Signed-off-by: Yoctopuce dev <dev@yoctopuce.com>
This option is needed for ports such as webassembly where objects are
proxied and can be identical without being the same C pointer.
Signed-off-by: Damien George <damien@micropython.org>
When this configuration flag is set, VfsPosix instances can be written.
Otherwise, they will always be created "read only".
This flag is useful when fuzzing micropython: Without VfsPosix, the fuzzing
input script cannot be read; but with writable VfsPosix, fuzzing scripts
can potentially perform undesired operations on the host filesystem.
Signed-off-by: Jeff Epler <jepler@gmail.com>
- DTLS spec recommends HelloVerify and Anti Replay protection be enabled,
and these are enabled in the default mbedTLS config. Implement them here.
- To help compensate for the possible increase in code size, add a
MICROPY_PY_SSL_DTLS build config macro that's enabled for EXTRA and
above by default.
This allows bare metal mbedTLS ports to use DTLS with HelloVerify support.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This is code makes sure that time functions work properly on a
reasonable date range, on all platforms, regardless of the epoch.
The suggested minimum range is 1970 to 2099.
In order to reduce code footprint, code to support far away dates
is only enabled specified by the port.
New types are defined to identify timestamps.
The implementation with the smallest code footprint is when
support timerange is limited to 1970-2099 and Epoch is 1970.
This makes it possible to use 32 bit unsigned integers for
all timestamps.
On ARM4F, adding support for dates up to year 3000 adds
460 bytes of code. Supporting dates back to 1600 adds
another 44 bytes of code.
Signed-off-by: Yoctopuce dev <dev@yoctopuce.com>
This commit introduces an optional feature to provide to native emitters
the fully qualified name of the entity they are compiling.
This is achieved by altering the generic ASM API to provide a third
argument to the entry function, containing the name of the entity being
compiled. Currently only the debug emitter uses this feature, as it is
not really useful for other emitters for the time being; in fact the
macros in question just strip the name away.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
When the symbol `__all__` is defined in a module, `mp_import_all()` should
import all listed symbols into the global environment, rather than relying
on the underscore-is-private default. This is the standard in CPython.
Each item is loaded in the same way as if it would be an explicit import
statement, and will invoke the module's `__getattr__` function if needed.
This provides a straightforward solution for fixing star import of modules
using a dynamic loader, such as `extmod/asyncio` (see issue #7266).
This improvement has been enabled at BASIC_FEATURES level, to avoid
impacting devices with limited ressources, for which star import is of
little use anyway.
Additionally, detailled reporting of errors during `__all__` import has
been implemented to match CPython, but this is only enabled when
ERROR_REPORTING is set to MICROPY_ERROR_REPORTING_DETAILED.
Signed-off-by: Yoctopuce dev <dev@yoctopuce.com>
This commit expands the Xtensa inline assembler to support most if not
all opcodes available on the ESP8266 and LX3 Xtensa cores.
This is meant as a stepping stone to add inline assembler support for
the ESP32 and its LX6 core, along to windowed-specific opcodes and
additional opcodes that are present only on the LX7 core (ESP32-S3 and
later).
New opcodes being added are covered by tests, and the provided tests
were expanded to also include opcodes available in the existing
implementation. Given that the ESP8266 space requirements are tighter
than ESP32's, certain opcodes that won't be commonly used have been put
behind a define to save some space in the general use case.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit renames the NORETURN macro, indicating to the compiler
that a function does not return, into MP_NORETURN to maintain the same
naming convention of other similar macros.
To maintain compaitiblity with existing code NORETURN is aliased to
MP_NORETURN, but it is also deprecated for MicroPython v2.
This changeset was created using a similar process to
decf8e6a8b ("all: Remove the "STATIC"
macro and just use "static" instead."), with no documentation or python
scripts to change to reflect the new macro name.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
IOBase is quite an important building block of other parts of the system,
such as `mpremote mount` and running .mpy and native tests.
This feature costs +244 bytes of firmware size on ARM Thumb2 architectures,
which is worth the cost for the extra features it enables.
The change here means that `io.IOBase` is now enabled on all nrf boards,
(previously it was only nRF52840 and nRF9160) and also B_L072Z_LRWAN1
(there is no change to other ports or boards).
Signed-off-by: Damien George <damien@micropython.org>
This is a pretty fundamental built-in and having CPython-compatible
behaviour is beneficial. The code size increase is not much, and
ports/boards can still disable it if needed to save space.
Addresses issue #5384.
Signed-off-by: Damien George <damien@micropython.org>
This is a generic interface to allow querying and modifying the read-only
memory area of a device, if it has such an area.
Signed-off-by: Damien George <damien@micropython.org>
This commit implements a small subset of the CPython `marshal` module. It
implements `marshal.dumps()` and `marshal.loads()`, but only supports
(un)marshalling code objects at this stage. The semantics match CPython,
except that the actual marshalled bytes is not compatible with CPython's
marshalled bytes.
The module is enabled at the everything level (only on the unix coverage
build at this stage).
Signed-off-by: Damien George <damien@micropython.org>
This allows retrieving the code object of a function using
`function.__code__`, and then reconstructing a function from a code object
using `FunctionType(code_object)`.
This feature is controlled by `MICROPY_PY_FUNCTION_ATTRS_CODE` and is
enabled at the full-features level.
Signed-off-by: Damien George <damien@micropython.org>
The `mp_obj_code_t` and `mp_type_code` code object was defined internally
in both `py/builtinevex.c` and `py/profile.c`, with completely different
implementations (the former very minimal, the latter quite complete).
This commit factors these implementations into a new, separate source file,
and allows the code object to have four different modes, selected at
compile-time:
- MICROPY_PY_BUILTINS_CODE_NONE: code object not included in the build.
- MICROPY_PY_BUILTINS_CODE_MINIMUM: very simple code object that just holds
a reference to the function that it represents. This level is used when
MICROPY_PY_BUILTINS_COMPILE is enabled.
- MICROPY_PY_BUILTINS_CODE_BASIC: simple code object that holds a reference
to the proto-function and its constants.
- MICROPY_PY_BUILTINS_CODE_FULL: almost complete implementation of the code
object. This level is used when MICROPY_PY_SYS_SETTRACE is enabled.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support for writing inline assembler functions when
targeting a RV32IMC processor.
Given that this takes up a bit of rodata space due to its large
instruction decoding table and its extensive error messages, it is
enabled by default only on offline targets such as mpy-cross and the
qemu port.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit defines a new ROMFS filesystem for storing read-only files that
can be memory mapped, and a new VfsRom driver. Files opened from this
filesystem support the buffer protocol. This allows naturally getting the
memory-mapped address of the file using:
- memoryview(file)
- uctypes.addressof(file)
Furthermore, if these files are .mpy files then their content can be
referenced in-place when importing. Such imports take up a lot less RAM
than importing from a normal filesystem. This is essentially dynamically
frozen .mpy files, building on the revamped v6 .mpy file format.
Signed-off-by: Damien George <damien@micropython.org>
This is necessary for mbedTLS callbacks that do not carry any user state,
so those callbacks can be customised per SSL context.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
This commit adds a new `RingIO` type which exposes the internal ring-buffer
code for general use in Python programs. It has the stream interface
making it similar to `StringIO` and `BytesIO`, except `RingIO` has a fixed
buffer size and is automatically safe when reads and writes are in
different threads or an IRQ.
This new type is enabled at the "extra features" ROM level.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Currently the stack limit margin is hard-coded in each port's call to
`mp_stack_set_limit()`, but on threaded ports it's fiddlier and can lead to
bugs (such as incorrect thread stack margin on esp32).
This commit provides a new API to initialise the C Stack in one function
call, with a config macro to set the margin. Where possible the new call
is inlined to reduce code size in thread-free ports.
Intended replacement for `MP_TASK_STACK_LIMIT_MARGIN` on esp32.
The previous `stackctrl.h` API is still present and unmodified apart from a
deprecation comment. However it's not available when the
`MICROPY_PREVIEW_VERSION_2` macro is set.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This is needed for a workaround on esp32 port (in child commit),
which produces incorrect results otherwise.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Being able to send data out in LSB format can be useful, and having support
in the low-level driver is much better than requiring Python code to
reorder the bits before sending them / after receiving them. In particular
if the hardware does not support the LSB format (eg RP2040) then one needs
to use the SoftSPI in LSB mode.
For this change a default definition of `MICROPY_PY_MACHINE_SPI_MSB/_LSB`
was added to `py/mpconfig.h`, making them available to all ports. The
identical defines in `esp32/mpconfigport.h` were deleted.
Resolves issues #5340, #11404.
Signed-off-by: robert-hh <robert@hammelrath.com>
The limit is set by a `MICROPY_PY_MACHINE_FREQ_NUM_ARGS_MAX` define, which
defaults to 1 and is set for stm32 to 4.
For stm32 this fixes a regression introduced in commit
e1ec6af654 where the maximum number of
arguments was changed from 4 to 1.
Signed-off-by: robert-hh <robert@hammelrath.com>
This emitter prints out pseudo-machine instructions, instead of the usual
output of the native emitter. It can be enabled on any port via
`MICROPY_EMIT_NATIVE_DEBUG` (make sure other native emitters are disabled)
but the easiest way to use it is with mpy-cross:
$ mpy-cross -march=debug file.py
Signed-off-by: Damien George <damien@micropython.org>
This adds a native code generation backend for RISC-V RV32I CPUs, currently
limited to the I, M, and C instruction sets.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
Allows passing in a callback to `TaskQueue()` that is called when something
is pushed on to the queue.
Signed-off-by: Damien George <damien@micropython.org>
Although the original motivation given for the workaround[1] is correct,
nlr.o and nlrthumb.o are linked with a small enough distance that the
problem does not occur, and the workaround isn't necessary. The distance
between the b instruction and its target (nlr_push_tail) is just 64
bytes[2], well within the ±2046 byte range addressable by an
unconditional branch instruction in Thumb mode.
The workaround induces a relocation in the text section (textrel), which
isn't supported everywhere, notably not on musl-libc[3], where it causes
a crash on start-up. With the workaround removed, micropython works on an
ARMv5T Linux system built with musl-libc.
This commit changes nlrthumb.c to use a direct jump by default, but
leaves the long jump workaround as an option for those cases where it's
actually needed.
[1]: commit dd376a239d
Author: Damien George <damien.p.george@gmail.com>
Date: Fri Sep 1 15:25:29 2017 +1000
py/nlrthumb: Get working again on standard Thumb arch (ie not Thumb2).
"b" on Thumb might not be long enough for the jump to nlr_push_tail so
it must be done indirectly.
[2]: Excerpt from objdump -d micropython:
000095c4 <nlr_push_tail>:
95c4: b510 push {r4, lr}
95c6: 0004 movs r4, r0
95c8: f02d fd42 bl 37050 <mp_thread_get_state>
95cc: 6943 ldr r3, [r0, #20]
95ce: 6023 str r3, [r4, #0]
95d0: 6144 str r4, [r0, #20]
95d2: 2000 movs r0, #0
95d4: bd10 pop {r4, pc}
000095d6 <nlr_pop>:
95d6: b510 push {r4, lr}
95d8: f02d fd3a bl 37050 <mp_thread_get_state>
95dc: 6943 ldr r3, [r0, #20]
95de: 681b ldr r3, [r3, #0]
95e0: 6143 str r3, [r0, #20]
95e2: bd10 pop {r4, pc}
000095e4 <nlr_push>:
95e4: 60c4 str r4, [r0, #12]
95e6: 6105 str r5, [r0, #16]
95e8: 6146 str r6, [r0, #20]
95ea: 6187 str r7, [r0, #24]
95ec: 4641 mov r1, r8
95ee: 61c1 str r1, [r0, #28]
95f0: 4649 mov r1, r9
95f2: 6201 str r1, [r0, #32]
95f4: 4651 mov r1, sl
95f6: 6241 str r1, [r0, #36] @ 0x24
95f8: 4659 mov r1, fp
95fa: 6281 str r1, [r0, #40] @ 0x28
95fc: 4669 mov r1, sp
95fe: 62c1 str r1, [r0, #44] @ 0x2c
9600: 4671 mov r1, lr
9602: 6081 str r1, [r0, #8]
9604: e7de b.n 95c4 <nlr_push_tail>
[3]: https://www.openwall.com/lists/musl/2020/09/25/4
Signed-off-by: J. Neuschäfer <j.ne@posteo.net>
Enabled by MICROPY_COMPILE_ALLOW_TOP_LEVEL_AWAIT. When enabled, this means
that scope such as module-level functions and REPL statements can yield.
The outer C code must then handle this yielded generator.
Signed-off-by: Damien George <damien@micropython.org>
This commit implements the 'e' half-float format: 10-bit mantissa, 5-bit
exponent. It uses native _Float16 if supported by the compiler, otherwise
uses custom bitshifting encoding/decoding routines.
Signed-off-by: Matthias Urlichs <matthias@urlichs.de>
Signed-off-by: Damien George <damien@micropython.org>
Add `pop()`, `appendleft()`, and `extend()` methods, support iteration
and indexing, and initializing from an existing sequence.
Iteration and indexing (subscription) have independent configuration flags
to enable them. They are enabled by default at the same level that
collections.deque is enabled (the extra features level).
Also add tests for checking new behavior.
Signed-off-by: Damien George <damien@micropython.org>
Disabled by default, but enabled on all boards that previously had
`MICROPY_PY_MACHINE_BARE_METAL_FUNCS` enabled.
Signed-off-by: Damien George <damien@micropython.org>
The STATIC macro was introduced a very long time ago in commit
d5df6cd44a. The original reason for this was
to have the option to define it to nothing so that all static functions
become global functions and therefore visible to certain debug tools, so
one could do function size comparison and other things.
This STATIC feature is rarely (if ever) used. And with the use of LTO and
heavy inline optimisation, analysing the size of individual functions when
they are not static is not a good representation of the size of code when
fully optimised.
So the macro does not have much use and it's simpler to just remove it.
Then you know exactly what it's doing. For example, newcomers don't have
to learn what the STATIC macro is and why it exists. Reading the code is
also less "loud" with a lowercase static.
One other minor point in favour of removing it, is that it stops bugs with
`STATIC inline`, which should always be `static inline`.
Methodology for this commit was:
1) git ls-files | egrep '\.[ch]$' | \
xargs sed -Ei "s/(^| )STATIC($| )/\1static\2/"
2) Do some manual cleanup in the diff by searching for the word STATIC in
comments and changing those back.
3) "git-grep STATIC docs/", manually fixed those cases.
4) "rg -t python STATIC", manually fixed codegen lines that used STATIC.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
