SPI Refactoring

[mrengineer7777]

@SuGlider @me-no-dev @PilnyTomas @VojtechBartoska
TLDR: I would love to chat with the maintainer for SPI, and any history I should be aware of before jumping into SPI refactoring.

I'm considering using a W5500 module for Ethernet support. I see a user has added support on #7163, but one of the major blockers is conflicts with the SPI bus. In fact, I see lots of issues linked to SPI. I'm considering a PR to start the refactoring myself. How hard could it be? Famous last words....
As you can see below I've already put a lot of time into researching SPI issues.

Goals:

• Refactor to IDF methods
• Non-breaking changes
• Transaction safety
• Thread safe
• Arduino API compatibility
• Easier usage of multiple SPI ports
• Add SPI Slave mode?

Dependecies
   Arduino "Ethernet" library uses SPI
   esp32-hal-spi.h
      esp32-hal.h
      SPI.h
   SPI.h
      SD_Test.ino
      SD_time.ino
      sd_diskio.h
      SD.h
      SMMC_time.ino

Things to watch out for:

• SPI is used extensively and has been confusing for many users
• SPI affects SD via SPI (SD.h)
• SPI varies significantly between ESP32 versions
• Is internal flash SPI driver completely independent of SPI library?

---Categorized issues---
Address Mask 0x3F
    655 https://github.com/espressif/arduino-esp32/issues/655  User issue
   1419 https://github.com/espressif/arduino-esp32/issues/1419  Expired no answer

Bugfixes
    640 https://github.com/espressif/arduino-esp32/issues/640  Added `spiTransferBits'.
        https://github.com/espressif/arduino-esp32/commit/ab0ea2f6da9ca842fbb3bce90179daeb3580d3da
   1094 https://github.com/espressif/arduino-esp32/issues/1094  Fixed SPI user.ck_out_edge value.
        https://github.com/espressif/arduino-esp32/pull/1096
   1623 https://github.com/espressif/arduino-esp32/issues/1623  Add Arduino's SPI.transfer(buffer, size) function.
        https://github.com/espressif/arduino-esp32/pull/2136
	  https://github.com/arduino-libraries/Ethernet/releases/tag/2.0.0  Uses SPI.transfer().
   1814 https://github.com/espressif/arduino-esp32/issues/1814  Documentation fix
        The ESP32 has four SPI buses, however as of right now only two of them are available to use, HSPI and VSPI.
           Simply using the SPI API as illustrated in Arduino examples will use VSPI, leaving HSPI unused.
   3241 https://github.com/espressif/arduino-esp32/issues/3241 SPI.writeBytes() definition different ESP32 vs ESP8266.
        https://github.com/espressif/arduino-esp32/pull/3242
   3730 https://github.com/espressif/arduino-esp32/pull/3730  Added SPI_HAS_TRANSACTION.
        https://github.com/espressif/arduino-esp32/commit/7af4490fcc2b9f549414921d9a8ad1392303f4d8
   4558 https://github.com/espressif/arduino-esp32/issues/4558  Fixed SPI.setHwCs().
        https://github.com/espressif/arduino-esp32/pull/4559
   4935 https://github.com/espressif/arduino-esp32/issues/4935
        https://github.com/espressif/arduino-esp32/commit/a31f30529d3411a0a4a6549c75aadd6e725e7f60
   5787 https://github.com/espressif/arduino-esp32/issues/5787  Changed SPIClass::transfer() signature to match Arduino API.
        https://github.com/espressif/arduino-esp32/pull/6734
        https://github.com/arduino/ArduinoCore-API/blob/e03b65374c614130aa1b11597e07b3b5089a726d/api/HardwareSPI.h#L109-L111
   6404 https://github.com/espressif/arduino-esp32/issues/6404  SPI thread safety.
        https://github.com/espressif/arduino-esp32/pull/6425
   7465 https://github.com/espressif/arduino-esp32/issues/7465  Wrover-kit SPI  Clarify documentation

Crashes
   1797 https://github.com/espressif/arduino-esp32/issues/1797  SPI crashes if user selects pins used by flash
   6601 https://github.com/espressif/arduino-esp32/issues/6601  Occassional lockup in spiWriteNL(). while(spi->dev->cmd.usr);

Custom SPI Driver
   2271 https://github.com/espressif/arduino-esp32/issues/2271  Custom SPI driver class.  Interferes with Arduino SPI functions.
   4195 https://github.com/espressif/arduino-esp32/issues/4195  SPI GPIO Matrix in IDF.  Custom user code.

Discussions
   https://github.com/espressif/arduino-esp32/discussions/6470  Custom SS lines
   https://github.com/espressif/arduino-esp32/discussions/6587  SPI.cpp sets _ss but never uses it
   https://github.com/espressif/arduino-esp32/discussions/6548  W5500 with SPI-ETH

Errata
   5137 https://github.com/espressif/arduino-esp32/issues/5137  ESP32 boot failure at power up due to GPIO12 invalid voltage.
        Solution is to burn eFuse.

Feature requests
   7063 https://github.com/espressif/arduino-esp32/issues/7063  Add QSPI mode. 3 parallel data streams with one clock.

Interrupt Safe?
   2000 https://github.com/espressif/arduino-esp32/issues/2000
   4295 https://github.com/espressif/arduino-esp32/issues/4295
   6404 https://github.com/espressif/arduino-esp32/issues/6404  Bugfix thread safety

Multiple SPI or alternate pins
   790  https://github.com/espressif/arduino-esp32/issues/790  VSPI & HSPI SPI buses simultaneously
        https://github.com/espressif/arduino-esp32/blob/master/libraries/SPI/examples/SPI_Multiple_Buses/SPI_Multiple_Buses.ino  VSPI + HSPI example for ESP32s3
   1219 https://github.com/espressif/arduino-esp32/issues/1219  ESP32 Use 2 SPI ports
        User example code  http://www.iotsharing.com/2017/05/how-to-use-arduino-esp32-to-store-data-to-sdcard.html
   2298 https://github.com/espressif/arduino-esp32/issues/2298  How to use HSPI with SPIClass
        When using the HSPI pins for interfacing SD card, the pinout says to connect MISO to GPIO12, but when it is connected we cannot upload the code since on GPIO 12 "boot fail if pulled high". So when we disconnect the MISO and upload the code, it doesn't work.
        Yes I was able to use the HSPI pins on ESP32, but uploading the code while MISO connected is still a problem for me. So when uploading the code, I disconnect the MISO jumper and when the code is uploaded, I connect it back.
   3572 https://github.com/espressif/arduino-esp32/issues/3572  Resolved
   5264 https://github.com/espressif/arduino-esp32/issues/5264  SPIClass cannot have more than one instance???
   5567 https://github.com/espressif/arduino-esp32/issues/5567  ESP32-S2 custom hardware. Unable to reproduce.
   5676 https://github.com/espressif/arduino-esp32/issues/5676  ESP32-S2 Doesn't work with SPI3.
        The final sketch snippet can be seen below, and works with SPI2 or SPI1 (FSPI & HSPI)
        It does not work with SPI3, and I'd love a solution as to why.
        Software SPI.  SPIClass spi = SPIClass(HSPI);
   5897 https://github.com/espressif/arduino-esp32/issues/5897  User error.  SPIClass spi = SPIClass(HSPI) cannot be local variable.
   5900 https://github.com/espressif/arduino-esp32/issues/5900  SPI HSPI with custom pins
   6252 https://github.com/espressif/arduino-esp32/issues/6252  SPIClass(HSPI) doesn't work but SPIClass(SPI) does.
   6372 https://github.com/espressif/arduino-esp32/issues/6372  SPIClass(VSPI) cannot be local variable.
   7543 https://github.com/espressif/arduino-esp32/issues/7543  ESP32-S3 Works with 1 SPI but not 2.
   7556 https://github.com/espressif/arduino-esp32/issues/7556  VSPI.  Dynamic configuration of pointer to SPI no longer works due to transaction lock?
   7596 https://github.com/espressif/arduino-esp32/issues/7596  ESP32-C3 Documentation of SPI channels may be inconsistent

Misc
    149 https://github.com/espressif/arduino-esp32/issues/149  SPI slow due to transaction locking
        There are generally two ways to speed things up:
           Update all libraries to use proper multi-byte api on ESP32 (Espressif is rolling out own ILI9341 driver based on Adafruit's API)
           Disable hardware locking by adding #define CONFIG_DISABLE_HAL_LOCKS 1 to esp32-hal.h
   1427 https://github.com/espressif/arduino-esp32/issues/1427  spi_struct_t definition hidden
        struct spi_struct_t definition hidden in .c file.  User defined struct in their code.

Refactoring
   6220 https://github.com/espressif/arduino-esp32/issues/6220 
   6597 https://github.com/espressif/arduino-esp32/issues/6597
   https://github.com/orgs/espressif/projects/3/views/5?pane=item&itemId=3266935
   https://github.com/orgs/espressif/projects/3/views/5?pane=item&itemId=4946329

SPI Slave Support.  Needs to integrate with SPI.h as SPI can only be master or slave.
    619 https://github.com/espressif/arduino-esp32/issues/619
        Add support for SPISlave mode like the ESP8266 SPISlave arduino library
            https://github.com/esp8266/Arduino/tree/master/libraries/SPISlave
            https://gist.github.com/shaielc/e0937d68978b03b2544474b641328145
        Failed PR 1413 https://github.com/espressif/arduino-esp32/pull/1413
        IDF SPI Slave: https://github.com/espressif/esp-idf/tree/master/examples/peripherals/spi_slave
        igrr Status update: we need to investigate how much we can adhere to the official SPISlave library API if we decide to implement this one.
    696 https://github.com/espressif/arduino-esp32/issues/696
        https://github.com/espressif/esp-idf/blob/master/examples/peripherals/spi_slave/receiver/main/app_main.c
   1654 https://github.com/espressif/arduino-esp32/issues/1654
   5442 https://github.com/espressif/arduino-esp32/issues/5442

Unresolved (won't fix?)
   1889 https://github.com/espressif/arduino-esp32/issues/1889 ESP32 SPI SS doesn't match Arduino
        arduino SPI.begin() documentation states SS pin should be driven to OUTPUT and set HIGH.  No action.

W5500 support
   5482 https://github.com/espressif/arduino-esp32/issues/5482  User code. Crashes on SPI.  https://github.com/mkrasselt1/esp32-lan-w5500
   5913 https://github.com/espressif/arduino-esp32/issues/5913  Recommended using Arduino Ethernet lib: https://www.arduino.cc/en/Reference/Ethernet

[me-no-dev]

@mrengineer7777 the SPI should not be refactored to use ESP-IDF's driver! We can accept fixes, but that is mostly it. The driver is thread-safe if transactions are used (which is the proper way). If we switch to ESP-IDF's SPI driver, we will experience huge slowdown on some uses and people will be very unhappy.

[VojtechBartoska]

thanks for nice research and suggestions @mrengineer7777. Great work, we will discuss this within a Team and let you know ASAP.

[mrengineer7777]

As you guys know I write clean fast code. A decrease in SPI performance would be terrible. I'm betting I can improve the library, but I also don't want to waste your time or mine if it won't be accepted. Looking forward to hearing what you guys decide.

[me-no-dev]

@mrengineer7777 you can always submit a PR with your proposed changes and we can see what could be improved :)

[mrengineer7777]

It appears IDF 5.0 makes significant changes to the SPI library. Perhaps we should wait on refactoring until IDF 5 is stable.

[Jason2866]

IDF 5.0 is stable. Probably you mean 5.1 ;-) where a lot of work is done. Don't expect it will be soon stable.
Latest version does not compile anymore with the adopted Arduino Lib Builder the work in progress branch from here.

[mrengineer7777]

@Jason2866 Thanks for the heads up. I'll probably wait on refactoring SPI until this repository moves to IDF 5.