freeHeap and vars

[MaBecker]

added some large libs and had to reduce vars to make Espruino run.

'variables' : 1100

>require("ESP8266").getState();
={
  sdkVersion: "2.2.1(6ab97e9)",
  cpuFrequency: 160, freeHeap: 7528, maxCon: 10,
  flashMap: "4MB:1024/1024",
  flashKB: 4096,
  flashChip: "0x1c 0x3016"
 }
>process.memory();
={ free: 1063, usage: 37, total: 1100, history: 11,
  gc: 0, gctime: 1.223 }
>

versus 'variables' : 1000

>require("ESP8266").getState();
={
  sdkVersion: "2.2.1(6ab97e9)",
  cpuFrequency: 160, freeHeap: 12784, maxCon: 10,
  flashMap: "4MB:1024/1024",
  flashKB: 4096,
  flashChip: "0x1c 0x3016"
 }
>process.memory();
={ free: 962, usage: 38, total: 1000, history: 17,
  gc: 0, gctime: 1.275 }
> 

Why is freeHeap 5256 larger when only using 1000 vars compared to 1100 vars?

[gfwilliams]

It's just http://www.espruino.com/Internals#variable-storage

If you get less than 1024 items then you need less bits for the address and it can use 12 bytes per var and not 16.

It's not all good since Strings are less efficient to store though.

[MaBecker]

Thanks.

Have you planed any optimization?

[gfwilliams]

I was toying with it (I think I have a branch here) - but if you don't go with 16 or 12 you mess up the alignment, which is obviously bad news for ESP8266/32. Also non-flat Strings get more inefficient, so it's not all good news.

[MaBecker]

let me try, there is CFLAGS += -mforce-l32 used by ESP8266_4MB for alignment

[gfwilliams]

https://github.com/espruino/Espruino/tree/experimental_compact_vars

It may well hurt performance though

[MaBecker]

Great - git clone and make flash - it works out of the box for a ESP8266_4MB build - many thanks!

 ____                 _
|  __|___ ___ ___ _ _|_|___ ___
|  __|_ -| . |  _| | | |   | . |
|____|___|  _|_| |___|_|_|_|___|
         |_| espruino.com
 2v01.64 (c) 2018 G.Williams
Espruino is Open Source. Our work is supported
only by sales of official boards and donations:
http://espruino.com/Donate
Flash map 4MB:1024/1024, manuf 0x1c chip 0x3016
>require("ESP8266").getState();
={
  sdkVersion: "2.2.1(6ab97e9)",
  cpuFrequency: 160, freeHeap: 18120, maxCon: 10,
  flashMap: "4MB:1024/1024",
  flashKB: 4096,
  flashChip: "0x1c 0x3016"
 }
>process.memory();
={ free: 1564, usage: 36, total: 1600, history: 6,
  gc: 0, gctime: 2.577 }
>

It may well hurt performance though

Well ESP8266 is running with 160MHz so should not hurt too much ;-)

Loaded some of my codes, worked without errors.

Any things you want me to test? Maybe some excessive memory stuff including timestamps to compare with master branch builds? Just name it and I try to test it.

[gfwilliams]

I'm not sure - it's worth checking that classes and flat Strings work ok though.

You did at one point have some method of running tests directly on the ESP8266? It might be worth trying a before and after and see if anything in particular breaks?

[MaBecker]

Ok, lets do some one device testing

Edit: Device Test Harness #1490

[wilberforce]

added some large libs and had to reduce vars to make Espruino run.

What large libs did you add - Why has this used heap space?

[MaBecker]

What large libs did you add

SHA256, SHA512, AES

Why has this used heap space?

good question ;-)

[wilberforce]

Try without AES. I added specific hacks to the linking phase to move the static tables for SHA1, SHA256 and SHA512 into RO so it goes to flash memory. I suspect the same think would need to happen for the AES code. What do you need AES for?

[MaBecker]

The goal is to build Espruino with a tiny TLS solution, will create a new issue for this soon.

Edit: Another goal: Add larger and/or color displays

[wilberforce]

I'm not sure that's achievable ;-(
Https setup requires ~ > 20,000 free heap to set up. We have run into an issue on the ESP32 with recent builds in that there is not enought free space to set up. With the ESP8266, you have even less memory so even if you get it to compile - I'm not sure you will have enough headroom to do the TLS setup.

#1613

[MaBecker]

I'm not sure that's achievable ;-(

It depends, like to check for with a minimal configuration for TLS 1.2 with PSK and AES-CCM ciphersuites like this config.h

[gfwilliams]

I'd posted in another bug (or was it email from @MaBecker ?) mbedtls allocates a 16kB packet buffer. If you control both ends of the connection then you can make that buffer way lower - I think mbed says 2k is ok.

Although obviously it may still be too tight. As @wilberforce says it'll probably require some hacks to get mbedtls arrays to go into flash rather than the usual ESP8266 thing of sticking them into RAM.

However I'm not too happy about screwing with every bit of Espruino and library it uses to force stuff that's already const into ROM. I'm pretty sure there was a way to get the compiler to behave normally and put const stuff into ROM? With the new compiler and latest flag that @MaBecker added I think it should now be able to do unaligned ROM reads, which would mean it's safe to work off of ROM?

[MaBecker]

As the major question is clearly answered, lets close this.