Page Title Image

Platform IO Coredumps with ESP32

When introducing threads and Mutex into our Leaf project, I’d introduced a deadlock. One that was super challenging to find by trying to think through the code. This is a writeup about how I used the Espressif IDF framework to troubleshoot my PlatformIO ESP32 Project.

Generating a Coredumps

The Arduino PlatformIO is basically the Espressif IDF framework in the back, with some less flexible, but generally sane defaults. The idea is that we can modify our lock guard to give up after 30 seconds, and throw an exception to generate a core dump so we can investigate the stack trace of all the threads.

Our platformIO file has the following lines:

[env]
platform = https://github.com/pioarduino/platform-espressif32/releases/download/stable/platform-espressif32.zip
board = esp32-s3-devkitc-1
framework = arduino
board_upload.flash_size = 8MB

# ~3.2Mb for A and B partitions to allow for updates
board_build.partitions = default_8MB.csv

This default_8MB.csv partition looks like this

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x330000,
app1,     app,  ota_1,   0x340000,0x330000,
spiffs,   data, spiffs,  0x670000,0x180000,
coredump, data, coredump,0x7F0000,0x10000,

The things to note here is the “coredump” partition defined with the coredump type in flash memory.

If we look deep into the core libraries, we can see a “sdkconfig.h” file with all of the FreeRTOS settings that would usually be configured using the “menuconfig” ESP-IDF settings.

Generated SDKConfig Generated SDKConfig Options

In here, we can tell that core dumping to flash is definitely enabled by default!

...
#define CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH 1
#define CONFIG_ESP_COREDUMP_DATA_FORMAT_ELF 1
#define CONFIG_ESP_COREDUMP_CHECKSUM_CRC32 1
#define CONFIG_ESP_COREDUMP_CHECK_BOOT 1
#define CONFIG_ESP_COREDUMP_ENABLE 1
#define CONFIG_ESP_COREDUMP_LOGS 1
#define CONFIG_ESP_COREDUMP_MAX_TASKS_NUM 64
#define CONFIG_ESP_COREDUMP_USE_STACK_SIZE 1
#define CONFIG_ESP_COREDUMP_STACK_SIZE 1792
...

I found I could download the coredump using a ‘pio pkg exec -p “tool-esptoolpy” — esptool.py’ command, but, that’s as far as I got without using the ESP-IDF tooling.

Installing ESP-IDF

I found stepping outside of the PlatformIO ecosystem and installing ESP-IDF the easiest way to access the tooling to download and parse these coredumps.

The installation guide has some setup instructions. Once the deps are installed:

git clone -b v5.2.5 --recursive https://github.com/espressif/esp-idf.git esp-idf
cd esp-idf
./install.sh esp32s3

Parsing a coredump

You’ll want to run the “export.sh” script (if bash) to setup the esp-idf environment variables first. Then we can point to the PIO built firmware.elf file.

In the dump below, we can tell that at the time of the crash, both the FanetRX, FanetTX, and the mainLoop’s display functions were all taking out GuardLocks. A great place to start :)

. ./export.sh 
[scott@sob-desktop esp-idf]$ espcoredump.py -p /dev/ttyACM0 info_corefile ~/local-scott/Documents/leaf/.pio/build/leaf_3_2_5_dev/firmware.elf 
WARNING: /home/scott/local-scott/Documents/leaf/.pio/build/leaf_3_2_5_dev/project_description.json does not exist. Please build the app with "idf.py build"
INFO: Invoke esptool to read image.
INFO: Retrieving core dump partition offset and size...
Traceback (most recent call last):
  File "/homes-local/scott/Documents/Others/esp-idf/components/partition_table/parttool.py", line 363, in <module>
    main()
    ~~~~^^
  File "/homes-local/scott/Documents/Others/esp-idf/components/partition_table/parttool.py", line 332, in main
    target = ParttoolTarget(**target_args)
  File "/homes-local/scott/Documents/Others/esp-idf/components/partition_table/parttool.py", line 103, in __init__
    self._call_esptool(['read_flash', str(partition_table_offset), str(gen.MAX_PARTITION_LENGTH), temp_file.name])
[scott@sob-desktop esp-idf]$ espcoredump.py -p /dev/ttyACM0 info_corefile ~/local-scott/Documents/leaf/.pio/build/leaf_3_2_5_dev/firmware.elf 
WARNING: /home/scott/local-scott/Documents/leaf/.pio/build/leaf_3_2_5_dev/project_description.json does not exist. Please build the app with "idf.py build"
INFO: Invoke esptool to read image.
INFO: Retrieving core dump partition offset and size...
INFO: Core dump partition offset=8323072, size=65536
WARNING: The core dump image offset is not specified. Use partition offset: 0x7f0000.
INFO: esptool.py v4.8.1
Serial port /dev/ttyACM0
Connecting...
Detecting chip type... ESP32-S3
Chip is ESP32-S3 (QFN56) (revision v0.2)
Features: WiFi, BLE, Embedded Flash 8MB (GD)
Crystal is 40MHz
MAC: f0:f5:bd:4f:ba:90
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
16 (100 %)
16 (100 %)
Read 16 bytes at 0x007f0000 in 0.0 seconds (30.0 kbit/s)...
Hard resetting via RTS pin...

INFO: esptool.py v4.8.1
Serial port /dev/ttyACM0
Connecting...
Detecting chip type... ESP32-S3
Chip is ESP32-S3 (QFN56) (revision v0.2)
Features: WiFi, BLE, Embedded Flash 8MB (GD)
Crystal is 40MHz
MAC: f0:f5:bd:4f:ba:90
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
32900 (100 %)
32900 (100 %)
Read 32900 bytes at 0x007f0000 in 2.9 seconds (91.9 kbit/s)...
Hard resetting via RTS pin...

===============================================================
==================== ESP32 CORE DUMP START ====================
The ROM ELF file won't load automatically since it was not found for the provided chip type.

Crashed task handle: 0x3fcc05dc, name: 'loopTask', GDB name: 'process 1070335452'
Crashed task is not in the interrupt context
Panic reason: assert failed: LockGuard::LockGuard(SemaphoreHandle_t) lock_guard.cpp:9 (0)

================== CURRENT THREAD REGISTERS ===================
exccause       0x1d (StoreProhibitedCause)
excvaddr       0x0
epc1           0x4037a8eb
epc2           0x0
epc3           0x421361a1
epc4           0x0
epc5           0x0
epc6           0x0
eps2           0x0
eps3           0x60020
eps4           0x0
eps5           0x0
eps6           0x0
pc             0x40377865          0x40377865 <panic_abort+21>
lbeg           0x40056f08          1074097928
lend           0x40056f12          1074097938
lcount         0x0                 0
sar            0x8                 8
ps             0x60620             394784
threadptr      <unavailable>
br             <unavailable>
scompare1      <unavailable>
acclo          <unavailable>
acchi          <unavailable>
m0             <unavailable>
m1             <unavailable>
m2             <unavailable>
m3             <unavailable>
expstate       <unavailable>
f64r_lo        <unavailable>
f64r_hi        <unavailable>
f64s           <unavailable>
fcr            <unavailable>
fsr            <unavailable>
a0             0x80383e8c          -2143797620
a1             0x3fcbfe70          1070333552
a2             0x3fcbfef4          1070333684
a3             0x1                 1
a4             0xa                 10
a5             0x60823             395299
a6             0x3fcaa948          1070246216
a7             0xabab              43947
a8             0x0                 0
a9             0x1                 1
a10            0x3fcbfebf          1070333631
a11            0x3fcbfebf          1070333631
a12            0xa                 10
a13            0x60023             393251
a14            0x3fcb6ac0          1070295744
a15            0xabab              43947

==================== CURRENT THREAD STACK =====================
#0  0x40377865 in panic_abort (details=0x3fcbfef4 "assert failed: LockGuard::LockGuard(SemaphoreHandle_t) lock_guard.cpp:9 (0)") at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:463
#1  0x40383e8c in esp_system_abort (details=0x3fcbfef4 "assert failed: LockGuard::LockGuard(SemaphoreHandle_t) lock_guard.cpp:9 (0)") at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/esp_system_chip.c:92
#2  0x4038af34 in __assert_func (file=<optimized out>, line=<optimized out>, func=<optimized out>, expr=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c:80
#3  0x4200a242 in LockGuard::LockGuard (this=0x3fcc0018, mutex=0x3fcb9880) at src/vario/lock_guard.cpp:9
#4  0x420065bc in FanetRadio::setCurrentLocation (this=0x3fca35f8 <FanetRadio::getInstance()::instance>, lat=@0x3fcc0058: 37.4380074, lon=@0x3fcc005c: -122.153625, alt=@0x3fcc0060: 27, heading=@0x3fcc0064: 153, climbRate=@0x3fcc0068: 0, speedKmh=@0x3fcc006c: 0.351880014) at src/vario/fanet_radio.cpp:305
#5  0x42007210 in FanetRadio::on_receive (this=0x3fca35f8 <FanetRadio::getInstance()::instance>, msg=...) at src/vario/fanet_radio.cpp:339
#6  0x42007a83 in etl::message_router<FanetRadio, GpsReading>::receive_message_type<GpsReading> (this=0x3fca35f8 <FanetRadio::getInstance()::instance>, msg=...) at .pio/libdeps/leaf_3_2_5_dev/Embedded Template Library/include/etl/message_router.h:519
#7  0x42007a98 in etl::message_router<FanetRadio, GpsReading>::receive (this=0x3fca35f8 <FanetRadio::getInstance()::instance>, msg=...) at .pio/libdeps/leaf_3_2_5_dev/Embedded Template Library/include/etl/message_router.h:450
#8  0x4200e8bc in etl::imessage_bus::receive (this=0x3fca6e04 <bus>, destination_router_id=<optimized out>, message=...) at .pio/libdeps/leaf_3_2_5_dev/Embedded Template Library/include/etl/message_bus.h:177
#9  0x4200e3f8 in etl::imessage_bus::receive (message=..., this=0x3fca6e04 <bus>) at .pio/libdeps/leaf_3_2_5_dev/Embedded Template Library/include/etl/message_bus.h:149
#10 MessageBus<(unsigned char)10>::receive (this=0x3fca6e04 <bus>, message=...) at src/vario/message_bus.h:24
#11 0x420085a9 in gps_read_buffer_once () at src/vario/gps.cpp:300
#12 0x4201204f in main_ON_loop () at src/vario/taskman.cpp:298
#13 0x4201207f in loop () at src/vario/taskman.cpp:195
#14 0x4206525b in loopTask (pvParameters=0x0) at /home/scott/.platformio/packages/framework-arduinoespressif32@src-36b172f5f4482da14324b9ce108a8e04/cores/esp32/main.cpp:74
#15 0x40384fdd in vPortTaskWrapper (pxCode=0x42065220 <loopTask(void*)>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

======================== THREADS INFO =========================
  Id   Target Id          Frame 
* 1    process 1070335452 0x40377865 in panic_abort (details=0x3fcbfef4 "assert failed: LockGuard::LockGuard(SemaphoreHandle_t) lock_guard.cpp:9 (0)") at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:463
  2    process 1070313456 0x40380fce in esp_cpu_wait_for_intr () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/cpu.c:64
  3    process 1070311988 0x40380fce in esp_cpu_wait_for_intr () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/cpu.c:64
  4    process 1070308848 0x400559e0 in ?? ()
  5    process 1070316960 prvProcessReceivedCommands () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/timers.c:856
  6    process 1070310272 0x400559e0 in ?? ()
  7    process 1070308480 0x400559e0 in ?? ()
  8    process 1070294804 0x400559e0 in ?? ()
  9    process 1070367764 0x400559e0 in ?? ()
  10   process 1070372244 0x400559e0 in ?? ()
  11   process 1070291812 0x400559e0 in ?? ()
  12   process 1070290404 0x400559e0 in ?? ()
  13   process 1070382832 0x400559e0 in ?? ()
  14   process 1070495392 0x400559e0 in ?? ()
  15   process 1070304488 0x400559e0 in ?? ()
  16   process 1070497752 0x400559e0 in ?? ()
  17   process 1070473668 0x400559e0 in ?? ()


       TCB             NAME PRIO C/B  STACK USED/FREE
---------- ---------------- -------- ----------------
0x3fcc05dc         loopTask    10/10       1808/14564
0x3fcbaff0            IDLE1      0/0          640/428
0x3fcbaa34            IDLE0      0/0          640/440
0x3fcb9df0          FanetRx      1/1        1744/2340
0x3fcbbda0          Tmr Svc      1/1         608/2496
0x3fcba380              tiT    18/18         752/3844
0x3fcb9c80          FanetTx      1/1         752/3332
0x3fcb6714  arduino_usb_eve      5/5         768/1276
0x3fcc8414          sys_evt    20/20         768/3836
0x3fcc9594   arduino_events    19/19         912/3180
0x3fcb5b64             ipc1    24/24          672/348
0x3fcb55e4             ipc0    24/24          672/348
0x3fccbef0             wifi    23/23         736/5904
0x3fce76a0      nimble_host    21/21         768/3312
0x3fcb8ce8        esp_timer    22/22         672/8024
0x3fce7fd8              BLE      9/9        1024/8964
0x3fce21c4     btController    23/23         736/3356

==================== THREAD 1 (TCB: 0x3fcc05dc, name: 'loopTask') =====================
#0  0x40377865 in panic_abort (details=0x3fcbfef4 "assert failed: LockGuard::LockGuard(SemaphoreHandle_t) lock_guard.cpp:9 (0)") at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:463
#1  0x40383e8c in esp_system_abort (details=0x3fcbfef4 "assert failed: LockGuard::LockGuard(SemaphoreHandle_t) lock_guard.cpp:9 (0)") at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/esp_system_chip.c:92
#2  0x4038af34 in __assert_func (file=<optimized out>, line=<optimized out>, func=<optimized out>, expr=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c:80
#3  0x4200a242 in LockGuard::LockGuard (this=0x3fcc0018, mutex=0x3fcb9880) at src/vario/lock_guard.cpp:9
#4  0x420065bc in FanetRadio::setCurrentLocation (this=0x3fca35f8 <FanetRadio::getInstance()::instance>, lat=@0x3fcc0058: 37.4380074, lon=@0x3fcc005c: -122.153625, alt=@0x3fcc0060: 27, heading=@0x3fcc0064: 153, climbRate=@0x3fcc0068: 0, speedKmh=@0x3fcc006c: 0.351880014) at src/vario/fanet_radio.cpp:305
#5  0x42007210 in FanetRadio::on_receive (this=0x3fca35f8 <FanetRadio::getInstance()::instance>, msg=...) at src/vario/fanet_radio.cpp:339
#6  0x42007a83 in etl::message_router<FanetRadio, GpsReading>::receive_message_type<GpsReading> (this=0x3fca35f8 <FanetRadio::getInstance()::instance>, msg=...) at .pio/libdeps/leaf_3_2_5_dev/Embedded Template Library/include/etl/message_router.h:519
#7  0x42007a98 in etl::message_router<FanetRadio, GpsReading>::receive (this=0x3fca35f8 <FanetRadio::getInstance()::instance>, msg=...) at .pio/libdeps/leaf_3_2_5_dev/Embedded Template Library/include/etl/message_router.h:450
#8  0x4200e8bc in etl::imessage_bus::receive (this=0x3fca6e04 <bus>, destination_router_id=<optimized out>, message=...) at .pio/libdeps/leaf_3_2_5_dev/Embedded Template Library/include/etl/message_bus.h:177
#9  0x4200e3f8 in etl::imessage_bus::receive (message=..., this=0x3fca6e04 <bus>) at .pio/libdeps/leaf_3_2_5_dev/Embedded Template Library/include/etl/message_bus.h:149
#10 MessageBus<(unsigned char)10>::receive (this=0x3fca6e04 <bus>, message=...) at src/vario/message_bus.h:24
#11 0x420085a9 in gps_read_buffer_once () at src/vario/gps.cpp:300
#12 0x4201204f in main_ON_loop () at src/vario/taskman.cpp:298
#13 0x4201207f in loop () at src/vario/taskman.cpp:195
#14 0x4206525b in loopTask (pvParameters=0x0) at /home/scott/.platformio/packages/framework-arduinoespressif32@src-36b172f5f4482da14324b9ce108a8e04/cores/esp32/main.cpp:74
#15 0x40384fdd in vPortTaskWrapper (pxCode=0x42065220 <loopTask(void*)>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 2 (TCB: 0x3fcbaff0, name: 'IDLE1') =====================
#0  0x40380fce in esp_cpu_wait_for_intr () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/cpu.c:64
#1  0x4206a144 in esp_vApplicationIdleHook () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/freertos_hooks.c:58
#2  0x40385e12 in prvIdleTask (pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:4353
#3  0x40384fdd in vPortTaskWrapper (pxCode=0x40385d94 <prvIdleTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 3 (TCB: 0x3fcbaa34, name: 'IDLE0') =====================
#0  0x40380fce in esp_cpu_wait_for_intr () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/cpu.c:64
#1  0x4206a144 in esp_vApplicationIdleHook () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/freertos_hooks.c:58
#2  0x40385e12 in prvIdleTask (pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:4353
#3  0x40384fdd in vPortTaskWrapper (pxCode=0x40385d94 <prvIdleTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 4 (TCB: 0x3fcb9df0, name: 'FanetRx') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fcb6ac0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x403846e5 in xQueueSemaphoreTake (xQueue=0x3fcb6a6c, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1853
#4  0x4200a22c in LockGuard::LockGuard (this=0x3fcc578c, mutex=0x3fcb6a6c) at src/vario/lock_guard.cpp:7
#5  0x4200e3ea in MessageBus<(unsigned char)10>::receive (this=0x3fca6e04 <bus>, message=...) at src/vario/message_bus.h:23
#6  0x4200759d in FanetRadio::processRxPacket (this=0x3fca35f8 <FanetRadio::getInstance()::instance>) at src/vario/fanet_radio.cpp:94
#7  0x420076b3 in FanetRadio::taskRadioRx (pvParameters=0x0) at src/vario/fanet_radio.cpp:46
#8  0x40384fdd in vPortTaskWrapper (pxCode=0x420076a0 <FanetRadio::taskRadioRx(void*)>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 5 (TCB: 0x3fcbbda0, name: 'Tmr Svc') =====================
#0  prvProcessReceivedCommands () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/timers.c:856
#1  prvTimerTask (pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/timers.c:688
#2  0x40384fdd in vPortTaskWrapper (pxCode=0x40384a4c <prvTimerTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 6 (TCB: 0x3fcba380, name: 'tiT') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fcba27c) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40384603 in xQueueReceive (xQueue=0x3fcba228, pvBuffer=0x3fcc6d8c, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x4208c942 in sys_arch_mbox_fetch (mbox=0x3fcab888 <tcpip_mbox>, msg=0x3fcc6d8c, timeout=500) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/port/freertos/sys_arch.c:317
#5  0x4208eec8 in tcpip_timeouts_mbox_fetch (mbox=<optimized out>, msg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:104
#6  tcpip_thread (arg=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:142
#7  0x40384fdd in vPortTaskWrapper (pxCode=0x4208ee30 <tcpip_thread>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 7 (TCB: 0x3fcb9c80, name: 'FanetTx') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fcb98d4) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x403846e5 in xQueueSemaphoreTake (xQueue=0x3fcb9880, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1853
#4  0x4200a22c in LockGuard::LockGuard (this=0x3fcc4b50, mutex=0x3fcb9880) at src/vario/lock_guard.cpp:7
#5  0x42006b58 in FanetRadio::taskRadioTx (pvParameters=0x0) at src/vario/fanet_radio.cpp:108
#6  0x40384fdd in vPortTaskWrapper (pxCode=0x42006a70 <FanetRadio::taskRadioTx(void*)>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 8 (TCB: 0x3fcb6714, name: 'arduino_usb_eve') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fcb5e30) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40384603 in xQueueReceive (xQueue=0x3fcb5ddc, pvBuffer=0x3fcb6540, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x4214e8b9 in esp_event_loop_run (event_loop=0x3fcb5db4, ticks_to_run=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_event/esp_event.c:644
#5  0x4214ea70 in esp_event_loop_run_task (args=0x3fcb5db4) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_event/esp_event.c:106
#6  0x40384fdd in vPortTaskWrapper (pxCode=0x4214ea64 <esp_event_loop_run_task>, pvParameters=0x3fcb5db4) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 9 (TCB: 0x3fcc8414, name: 'sys_evt') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fcc6f80) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40384603 in xQueueReceive (xQueue=0x3fcc6f2c, pvBuffer=0x3fcc8240, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x4214e8b9 in esp_event_loop_run (event_loop=0x3fcb9f98, ticks_to_run=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_event/esp_event.c:644
#5  0x4214ea70 in esp_event_loop_run_task (args=0x3fcb9f98) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_event/esp_event.c:106
#6  0x40384fdd in vPortTaskWrapper (pxCode=0x4214ea64 <esp_event_loop_run_task>, pvParameters=0x3fcb9f98) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 10 (TCB: 0x3fcc9594, name: 'arduino_events') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fcba544) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40384603 in xQueueReceive (xQueue=0x3fcba4f0, pvBuffer=0x3fcc9320, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x42121d8e in NetworkEvents::_checkForEvent (this=0x3fcb3cc8 <Network+4>) at /home/scott/.platformio/packages/framework-arduinoespressif32@src-36b172f5f4482da14324b9ce108a8e04/libraries/Network/src/NetworkEvents.cpp:111
#5  0x42121db0 in operator() (self=0x3fcb3cc8 <Network+4>, __closure=0x0) at /home/scott/.platformio/packages/framework-arduinoespressif32@src-36b172f5f4482da14324b9ce108a8e04/libraries/Network/src/NetworkEvents.cpp:66
#6  _FUN () at /home/scott/.platformio/packages/framework-arduinoespressif32@src-36b172f5f4482da14324b9ce108a8e04/libraries/Network/src/NetworkEvents.cpp:67
#7  0x40384fdd in vPortTaskWrapper (pxCode=0x42121da8 <_FUN(void*)>, pvParameters=0x3fcb3cc8 <Network+4>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 11 (TCB: 0x3fcb5b64, name: 'ipc1') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fc9d870 <xKernelLock>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40387301 in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:5765
#4  0x40376a79 in ipc_task (arg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_ipc.c:62
#5  0x40384fdd in vPortTaskWrapper (pxCode=0x40376a38 <ipc_task>, pvParameters=0x1) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 12 (TCB: 0x3fcb55e4, name: 'ipc0') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fc9d870 <xKernelLock>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40387301 in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:5765
#4  0x40376a79 in ipc_task (arg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_ipc.c:62
#5  0x40384fdd in vPortTaskWrapper (pxCode=0x40376a38 <ipc_task>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 13 (TCB: 0x3fccbef0, name: 'wifi') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fcc9e1c) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40384603 in xQueueReceive (xQueue=0x3fcc9dc8, pvBuffer=0x3fccbd40, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x420a676c in queue_recv_wrapper (queue=0x3fcc9dc8, item=0x3fccbd40, block_time_tick=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_wifi/esp32s3/esp_adapter.c:317
#5  0x42148a80 in ppTask ()
#6  0x40384fdd in vPortTaskWrapper (pxCode=0x42148a54 <ppTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 14 (TCB: 0x3fce76a0, name: 'nimble_host') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fce63e8) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40384603 in xQueueReceive (xQueue=0x3fce6394, pvBuffer=0x3fce74b8, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x4204f55c in npl_freertos_eventq_get (evq=0x3fca8be4 <g_eventq_dflt>, tmo=4294967295) at .pio/libdeps/leaf_3_2_5_dev/NimBLE-Arduino/src/nimble/porting/npl/freertos/src/npl_os_freertos.c:1469
#5  0x40375822 in ble_npl_eventq_get (tmo=4294967295, evq=0x3fca8be4 <g_eventq_dflt>) at .pio/libdeps/leaf_3_2_5_dev/NimBLE-Arduino/src/nimble/porting/npl/freertos/include/nimble/nimble_npl_os.h:512
#6  nimble_port_run () at .pio/libdeps/leaf_3_2_5_dev/NimBLE-Arduino/src/nimble/porting/nimble/src/nimble_port.c:311
#7  0x4203b28d in NimBLEDevice::host_task (param=0x0) at .pio/libdeps/leaf_3_2_5_dev/NimBLE-Arduino/src/NimBLEDevice.cpp:835
#8  0x40384fdd in vPortTaskWrapper (pxCode=0x4203b284 <NimBLEDevice::host_task(void*)>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 15 (TCB: 0x3fcb8ce8, name: 'esp_timer') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fc9d870 <xKernelLock>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40387301 in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:5765
#4  0x4206db3c in timer_task (arg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer.c:480
#5  0x40384fdd in vPortTaskWrapper (pxCode=0x4206db2c <timer_task>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 16 (TCB: 0x3fce7fd8, name: 'BLE') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fce7d40) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x40384603 in xQueueReceive (xQueue=0x3fce7cec, pvBuffer=0x3fcebe3c, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4  0x420054a5 in BLE::bleTask (args=0x3fca35a0 <BLE::get()::instance>) at src/vario/ble.cpp:154
#5  0x40384fdd in vPortTaskWrapper (pxCode=0x42005468 <BLE::bleTask(void*)>, pvParameters=0x3fca35a0 <BLE::get()::instance>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139

==================== THREAD 17 (TCB: 0x3fce21c4, name: 'btController') =====================
#0  0x400559e0 in ?? ()
#1  0x4038527a in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2  vPortExitCritical (mux=0x3fce111c) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3  0x403846e5 in xQueueSemaphoreTake (xQueue=0x3fce10c8, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1853
#4  0x420a7119 in semphr_take_wrapper ()
#5  0x4037ac34 in btdm_controller_task ()
#6  0x40384fdd in vPortTaskWrapper (pxCode=0x4037aac8 <btdm_controller_task>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139


======================= ALL MEMORY REGIONS ========================
Name   Address   Size   Attrs
.rtc.text 0x600fe000 0x100 R XA
.rtc.force_fast 0x600fe100 0x8 RW A
.rtc_noinit 0x50000200 0x0 RW  
.rtc.force_slow 0x50000200 0x0 RW  
.iram0.vectors 0x40374000 0x403 R XA
.iram0.text 0x40374404 0x18b13 R XA
.dram0.data 0x3fc9d000 0x6094 RW A
.flash.text 0x42000020 0x151a56 R XA
.flash.appdesc 0x3c160020 0x100 R  A
.flash.rodata 0x3c160120 0x6adc0 RW A
.iram0.data 0x4038d000 0x0 RW  
.iram0.bss 0x4038d000 0x0 RW  
.dram0.heap_start 0x3fcb4240 0x0 RW  
.coredump.tasks.data 0x3fcc05dc 0x160 RW 
.coredump.tasks.data 0x3fcbfdb0 0x810 RW 
.coredump.tasks.data 0x3fcbaff0 0x160 RW 
.coredump.tasks.data 0x3fcbad50 0x280 RW 
.coredump.tasks.data 0x3fcbaa34 0x160 RW 
.coredump.tasks.data 0x3fcba7a0 0x280 RW 
.coredump.tasks.data 0x3fcb9df0 0x160 RW 
.coredump.tasks.data 0x3fcc5630 0x6d0 RW 
.coredump.tasks.data 0x3fcbbda0 0x160 RW 
.coredump.tasks.data 0x3fcbbb20 0x260 RW 
.coredump.tasks.data 0x3fcba380 0x160 RW 
.coredump.tasks.data 0x3fcc6c20 0x2f0 RW 
.coredump.tasks.data 0x3fcb9c80 0x160 RW 
.coredump.tasks.data 0x3fcc4a00 0x2f0 RW 
.coredump.tasks.data 0x3fcb6714 0x160 RW 
.coredump.tasks.data 0x3fcb6400 0x300 RW 
.coredump.tasks.data 0x3fcc8414 0x160 RW 
.coredump.tasks.data 0x3fcc8100 0x300 RW 
.coredump.tasks.data 0x3fcc9594 0x160 RW 
.coredump.tasks.data 0x3fcc91f0 0x390 RW 
.coredump.tasks.data 0x3fcb5b64 0x160 RW 
.coredump.tasks.data 0x3fcb58b0 0x2a0 RW 
.coredump.tasks.data 0x3fcb55e4 0x160 RW 
.coredump.tasks.data 0x3fcb5330 0x2a0 RW 
.coredump.tasks.data 0x3fccbef0 0x160 RW 
.coredump.tasks.data 0x3fccbbf0 0x2e0 RW 
.coredump.tasks.data 0x3fce76a0 0x160 RW 
.coredump.tasks.data 0x3fce7380 0x300 RW 
.coredump.tasks.data 0x3fcb8ce8 0x160 RW 
.coredump.tasks.data 0x3fcb8a30 0x2a0 RW 
.coredump.tasks.data 0x3fce7fd8 0x160 RW 
.coredump.tasks.data 0x3fcebd00 0x400 RW 
.coredump.tasks.data 0x3fce21c4 0x160 RW 
.coredump.tasks.data 0x3fce1ed0 0x2e0 RW 

===================== ESP32 CORE DUMP END =====================
===============================================================
Done!