Core library upgrade

[proddy]

Following up from #1804, after 3.7.0 is out look at migrating some of the core libraries and frameworks to see if it improves performance, memory consumption and the firmware binary filesize isn't too big.

• https://github.com/pioarduino/platform-espressif32 to use Arduino v3.1/IDF v5.3 (from v2.0.x/IDF v4.4.x)
• Web Server to https://github.com/mathieucarbou/ESPAsyncWebServer and then PsychicHTTP with SSL when it's mature
• MQTT client espmqttclient -> https://github.com/mlesniew/PicoMQTT (no more AsyncTCP)

[MichaelDvP]

The factory partition for 16M is already nearly full. When creating it the idea was to put only a small loader in, not the full emsesp as now. So better use symmetrical partition sizes (3.6.5 have different partitions, so it fits in the braking change of 3.7.0 if we change now). Should also be used in emsesp-flasher.
Suggestion:

# Name,   Type, SubType, Offset,   Size,    Flags
nvs,      data, nvs,     0x9000,   0x005000,
otadata,  data, ota,     ,         0x002000,
boot,     app,  factory, ,         0x480000,
app0,     app,  ota_0,   ,         0x490000,
app1,     app,  ota_1,   ,         0x490000,
nvs1,     data, nvs,     ,         0x040000,
spiffs,   data, spiffs,  ,         0x200000,
coredump, data, coredump,,         0x010000,

[proddy]

Are you suggesting we keep boot smaller and put in https://github.com/emsesp/EMS-ESP-Loader ? I'm fine with that. Now is the time for breaking changes.

It would mean changing a lot though, from platform.io, the bbqkees flash too, the Flash Tool and the Web Installer. And we probably need to put back in the WebUI the option to restart another partition.

PIO always uploads to boot but doesn't adjust the partition info, so sometimes, when you restart after and esptool.py, it will go back to the old firmware version. I think this is a bug in platformio that Jason2866 just fixed in pioarduino.

I've updated the notes I keep on the partitions here. Can you review it please?
https://github.com/emsesp/bbqkees_flash/blob/main/README.md#partition-explanation

(not to others; this is private repo so you may not have access)

[MichaelDvP]

Are you suggesting we keep boot smaller and put in https://github.com/emsesp/EMS-ESP-Loader ?

No, i suggust to use normal emsesp and make boot nearly same size as ota (now boot is 280000, i want to enlarge to 480000).
The loader is only usefull if we ran out of mem in 4M chips.

[proddy]

ok. why not make all app sizes the same size? This change will erase the nvs data (nrg's and bbqkees's codes). BBQKees codes should be ok as this is not yet in production. The energy counters, is this going to hurt people? If so, maybe make export to Json in the WebUI and upload. The settings will also go, but they can be backed up. I do think we need to do this asap in 3.7.0. And I really want 3.7.0 out in the next weeks as I'm going to be busy at work for the next 6 months

[MichaelDvP]

ok. why not make all app sizes the same size?

It's nearly same size (64k diff on 4,7M partition), but i don't want to move nvs1/spiffs and keep sizes/bounderies on round values.

[proddy]

ah I see. Ok, let's do it.

[proddy]

btw I tried the latest pioarduino 3.0.7 and used heap went down from 208->196, max alloc too, and bin size +280KB so I think we'll stick with Arduino 2.x until 3.1 comes out.

[MichaelDvP]

With pioarduino the network scan still is broken: "no networks found"

[proddy]

With pioarduino the network scan still is broken: "no networks found"

should be fixed in the 3.1.0 - https://github.com/espressif/arduino-esp32/issues/ 10281 hopefully

more info at https://discord.com/channels/1263397951829708871/1263397951829708874/1287027557677662261

[proddy]

We can try Arduino Core 3.1 and ESP-IDF 5.3 with

platform = https://github.com/pioarduino/platform-espressif32/releases/download/53.03.10-rc2/platform-espressif32.zip

but will crash without patching NTP and AsyncTCP. See https://github.com/espressif/arduino-esp32/issues/ 10526