Monday, 16 November 2015

ESP8266 and Arduino for Logging Data to the Web

Update 2016-12-30: The process has become much simpler. Use the ThingSpeak or AdafruitIO libraries available for IDE 1.6.x and above to interact and avoid having to assemble your own network communications. ThingSpeak is free for the first 3 million channel updates a year, then paid. AdafruitIO is free, but you are limited to 10 fields total. Both limit the update speeds.

One of my students (Stephane Leahy) insisted I should find a way to include the ESP8266 in my grad measurements course for next term, so I had to give it a try. Now that I've proven I can do it, I'm not sure putting it in a course would be so useful, as this is bound to be a moving target. I think the best advice is to know this is possible, then go looking for a sensible solution when you have an actual application that needs it. The ESP8266 is a really cheap solution to adding WiFi when you need it, and may be enough micro-controller to stand alone if you don't need a lot of analog input.

I bought an Adafruit Huzzah ESP8266 breakout board and combined it with an Arduino Micro and some sensors. I used some sample code Stephane provided, then built on it to overcome some of the difficulties I ran into. Wiring was straightforward following the instructions. I added a TMP36 connected to the A (analog A0) pin on the 8266 and connected the RX/TX pins to listen to output from the Micro.


The 8266 is programmed with an FTDI cable over the serial port, so the micro needs to be unpowered, or at least silent, while the programming the 8266. I sorted this out by using a diode to bridge the two power busses at the top of the board. (It's hiding behind the blue pot.) When the Micro is plugged in, it will power both, with the 8266 getting about 4.1V. When only the 8266 is plugged in the Micro is unpowered and silent. I used the Arduino IDE to program the 8266 because it was familiar. When live, the Micro was just chugging along in data logger mode, spitting out lines of CSV data every second or so to the serial port that looked like this:

477971510,   734,   719,   719, 11001111,  1500,  1466,  1461,  3

The 8266 reads this data, parses it, reads an analog input and converts it to temperature, then sends all of that off to ThingSpeak for logging and reporting in the cloud. The resulting graphs look like this once things work:

This particular sequence shows how measuring temperature requires a little finesse. Sitting in the dark all night, the temperature was stable around 22.5C, but once I started working the temperature around my desk started rising. Unplugging the breadboard let it cool down, only to heat back up when it was plugged in again. Finally around 1500 I moved the board to a windowsill to catch the last few rays of late November sun. The indicated temperature peaked just as the sun went behind the building next door. It's hard to tell what is actual room temperature rise through the day and what is just sensor heating from the electronics or the sun.

ESP8266 Challenges

Analog Reading Scale: It wasn't obvious to me what the 8266 was using as a voltage reference for the analog conversions, so I used my multimeter to measure the conversion constant. The lack of clear documentation for the chip is a significant failing, observed by plenty of others.

The fiddly little buttons need to be actuated in sequence, so it was difficult for me to start the programming sequence with my full size fingers.

Analog disables connection for about 5 seconds for no apparent reason (see above for lack of documentation). This quirk would be enough to make me avoid using the analog input entirely, especially since there's only the one channel.

Retrying connection can crash the whole system with an exception number 9, which the web seems to think has something to do with too many connections open, or something (docs?).

Power hungry is only an issue if you need to run the chip on batteries for an extended period.

The code below should be reasonably readable to show how I got past these hurdles. It took a few days to track down the random exceptions, but I will keep my fingers crossed. Feel free to copy and play around with it. Good Luck!


// a minimal ThingSpeak logger based on Stephane Leahy's code --- Rick Sellens 2015-11-14
#include <ESP8266WiFi.h>

// this file should include the next few lines, but with your ssid, password, and thingspeak channel API key
#include "ssid.h"   // create ssid.h as another file tab in your Arduino IDE project.
// #define SSID "MyNetworkName" 
// #define PASS "MyCleverPassword"
// #define HOST "api.thingspeak.com"   // usually results in 0 or 1 retries to get a connection but takes longer then IP
// String link = "/update?key=TheAPIKeyFromMyThingSpeakChannel";

/*
The first client.connect(HOST,80) after reboot always succeeds with 0 retries.
Defining HOST as numerical and commenting out the analogRead() and delay(5000) connects first time, every time.
Uncommenting the analogRead() and success requires about 500 retries (about 5 seconds at 10 ms each).
Switching to a domain name for HOST succeeds after 1 retry (about 5 seconds).
Uncommenting the delay(5000) succeeds with 0 retries for domain name and 0 or 1 retries for numerical IP.
Making the delay 7000 succeeds with 0 retries most times for the numerical IP.

Behaviour is the same when HOST is 205.189.10.43 for weather.gc.ca, so it seems to be a local issue
Moral is probably to accept failure for a while..... and use the domain name version

The logging sometimes fails, so I added a restart condition if the connection doesn't get made quickly enough.
The latest failure said "ets Jan  8 2013,rst cause:4, bootmode:(1,7) \n\n wdt reset" which the online people
suggest has to do with a watchdog timer reset that may happen if it gets stuck in a loop for more than a second,
so it seems like a good idea to make the serial timeout shorter than a second to see if that makes the problem go away.
It didn't go away
  
 870942170,   4998,    477,    343, 0, 1, 1, 1, 1, 1, 1, 0,  10230,    970,    720, 
 871442210,   4998,    476,    346, 0, 1, 1, 1, 1, 1, 1, 0,  10230,    970,    721, 
 871942210,   4998,    476,    348, 0, 1, 1, 1, 1, 1, 1, 0,  10230,    970,    721, 
 872442250,   4998,    477,    344, 0, 1, 1, 1, 1, 1, 1, 0,  10230,    972,    673, 
 872942280,   4998,    477,    341, 0, 1, 1, 1, 1, 1, 1, 0,  10230,    972,    673, 
 873442310,   4998,    477,    338, 0, 1, 1, 1, 1, 1, 1, 0,  10230,    972,    671, 
 873942350,   4998,    477,    336, 0, 1, 1, 1, 1, 1, 1, 0,  10230,    973,    668, 
 874442390,   4998,    477,    338, 0, 1, 1, 1, 1, 1, 1, 0,  10230,    970,    701, 
 874942420,   4998,    477,    340, 0, 1, 1, 1, 1, 1, 1, 0,  10230,    970,    703, 
 875442450, 875442432, 875442432, 875442432, 875442432, 875442432, 875442432, 875442432, 8, 7, 5, 4, 4, 2, 4, 5, 0, 875442450,   4998,    476,    342, 1111110,  10230,    970, 
Exception (9):
epc1=0x401018af epc2=0x00000000 epc3=0x00000000 excvaddr=0x202c3029 depc=0x00000000

ctx: cont 
sp: 3ffea5f0 end: 3ffea870 offset: 01a0

>>>stack>>>
3ffea790:  ffffffff 00000007 3ffea8c8 40202541  
3ffea7a0:  3738200a 32343435 2c303534 35373820  
3ffea7b0:  34323434 202c3233 34353738 33343234  
3ffea7c0:  38202c32 34343537 32333432 3738202c  
3ffea7d0:  32343435 2c323334 35373820 34323434  
3ffea7e0:  202c3233 34353738 33343234 38202c32  
3ffea7f0:  34343537 32333432 2c38202c 202c3720  
3ffea800:  34202c35 2c34202c 202c3220 35202c34  
3ffea810:  2c30202c 35373820 34323434 202c3035  
3ffea820:  39342020 202c3839 34202020 202c3637  
3ffea830:  33202020 202c3234 31313131 2c303131  
3ffea840:  30312020 2c303332 20202020 2c303739  
3ffea850:  3fff0020 00000000 3ffea894 4020186e  
3ffea860:  00000000 00000000 3ffe9850 40100378  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(1,7)


 ets Jan  8 2013,rst cause:4, boot mode:(1,7)

wdt reset

Apparently exception 9 has something to do with too many floating clients, or something, so I tried lengthening the time
between connection attempts from 10 ms to 100 ms, so it doesn't have to try as many times before the 5 s runs out, or whatever.
2015-11-16: Then it ran all night. 
 */


float smoothTemp = 20.0;
unsigned long t = 0;

void setup() {
  Serial.begin(115200);
  Serial.setTimeout(500L);  // timeout in half a second to keep the wdt reset from triggering on the ESP8266

  // connect to WiFi
  WiFi.begin(SSID, PASS);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("WiFi connected with ");
  Serial.print("IP address ");
  Serial.println(WiFi.localIP());

}

static unsigned raw[] = {0,0,0,0,0,0,0,0,0,0};
static float cal[] = {0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0};
static byte digio[] = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};

void loop() {
  static unsigned connFail = 0;       // number of times the connection has failed
  static unsigned smoothies = 0;      // number of samples that have fed into the smoothing, or 20000 max
  static unsigned reported = 0;       // set to one when new data comes in, then back to zero when logged
  unsigned data = 750;                // a not-stupid, not-zero value that could have come from the analog input

  // serial incoming lines from the logger like:  477971510,   734,   719,   719, 11001111,  1500,  1466,  1461,  3
  if(Serial.available()){                            // only if there's something waiting
    String line = Serial.readStringUntil('\n');
    //parse the string for data to report
    String s = line.substring(line.length()-1);     // last character is the number of channels        
    unsigned n = s.toInt();
    line = line.substring(0,line.lastIndexOf(","));

    // take n channels of raw values off the end of the string
    for(int i = n-1;i>=0;i--){
      s = line.substring(line.lastIndexOf(",")+1);
      raw[i] = s.toInt();
      line = line.substring(0,line.lastIndexOf(","));
    }
    
    // take the digital data off the end of the string
    s = line.substring(line.lastIndexOf(",")+1);
    s.trim();
    unsigned nd = s.length();
    for(int i = 0;i < nd;i++) digio[i] = s.charAt(i) - '0';
    line = line.substring(0,line.lastIndexOf(","));

    // take n channels of calibrated data off the end of the string
    for(int i = n-1;i>=0;i--){
      s = line.substring(line.lastIndexOf(",")+1);
      cal[i] = s.toFloat();
      line = line.substring(0,line.lastIndexOf(","));
    }

    // take the time since reboot in us off the end of the string
    line = line.substring(0,line.length() - 1);       // drop a digit to fit inside the long int limits
    unsigned ts = (unsigned long) line.toInt() * 10;  // put the digit back on to read rounded down to 10
    
    // report as a check.
    char scratchy[100];
    sprintf(scratchy,"\n%10lu, ",ts);
    for(int i = 0; i < n; i++) sprintf(scratchy,"%s%6u, ",scratchy,(unsigned) cal[i]);
    for(int i = 0; i < nd; i++) sprintf(scratchy,"%s%1u, ",scratchy,(unsigned) digio[i]);
    for(int i = 0; i < n; i++) sprintf(scratchy,"%s%6u, ",scratchy,raw[i]);
    if(connFail == 0) Serial.print(scratchy);
    
    // ditch any leftovers
    while (Serial.available()){ 
      char c = Serial.read();
    }
    reported++;     // some data got reported on the Serial port
  }
  
  data = analogRead(A0);
  if(data != 0){
    float mvData = data /1.056;   // constant is empirically measured
    smoothTemp = smoothTemp *.99 + ((mvData - 500.) / 10.)*.01;
  }
  if (smoothies < 20000) smoothies++;

  // This part should be last in the loop, since it will generate an early return on failure.
  // if( long enough to have valid data && (it has been long enough || this is the first time) ) then log data
  if ( smoothies > 10000 && ((millis() - t > 1000 * 20L) || (t == 0)) ) {
    String url = link + "&field1=" + smoothTemp;
    // report the Serial data if it has been updated a few times since last logging -- ignore plug/unplug
    if(reported > 5) url = url + "&field2=" + (cal[0]/1000) + "&field3=" + (cal[1]/1000)   + "&field4=" + (cal[2]/1000) 
                      + "&field5=" + (1.6 + 0.8 * digio[2])  
                      + "&field6=" + (4.6 + 0.8 * digio[5]) 
                      + "&field7=" + (6.6 + 0.8 * digio[7]);
    WiFiClient client;            // create TCP connection
    if(connFail == 0){ Serial.print("\nOpening connection to: ");  Serial.print(HOST); }
    //delay(5000);                // this delay may let things settle long enough to get a connection first time
    if (!client.connect(HOST, 80)) {    // this connection seems to fail often, except the first try after reboot
      if (connFail == 0) Serial.print(" failed ... ");
      else if(connFail%100 == 0) Serial.print(".");
      connFail++;
      delay(200);             // don't retry immediately -- avoid flooding net connection and ESP exception 9
      if(connFail > 100){     // redo from start if it has taken too long
        Serial.print("Making a clean start of it\n\n");
        WiFi.disconnect();
        delay(1000);
        setup();
        connFail = 0;
      }
      return;       // don't even bother with the GET
    }
    // send GET request to the server, read reply and print it
    Serial.print(" success after "); Serial.print(connFail); Serial.print(" retries!\nRequesting URL: ");
    connFail = 0;
    Serial.print(url); Serial.print(" ... ");
    client.print(String("GET ") + url + " HTTP/1.1\r\n" + "Host: " + HOST + "\r\n" + "Connection: close\r\n\r\n");
    reported = 0;       // the data just got logged
    // wait up to 1000 ms to hear back from the server, then move on
    for(unsigned long waiting = millis(); !client.available() && millis()-waiting < 1000;);
    //while (client.available()) {      // uncomment the while if you want more than just one line
      String line = client.readStringUntil('\r');
      Serial.print(line);
    //}
    t = millis();
  }

}

Wednesday, 28 October 2015

Serial Buffer Sizes on Arduino

Often you want to leave the serial port alone to send or receive data under interrupts while the main loop of the code wanders off to do other things. If you don't get back often enough, the buffer may overflow. The standard buffer size in the Arduino IDE is 64 bytes (or only 16 for really small RAM boards). The code is in arduino/avr/cores/arduino/HardwareSerial.h and looks something like this:

#if !defined(SERIAL_TX_BUFFER_SIZE)
#if (RAMEND < 1000)
#define SERIAL_TX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 64
#endif
#endif

Altering that code to something like this can get you some more buffer space on larger processors.

#if !defined(SERIAL_RX_BUFFER_SIZE)
#if (RAMEND < 1000)
#define SERIAL_RX_BUFFER_SIZE 16
#else
// this test will create a large buffer on anything big
#if (RAMEND > 2200)
#define SERIAL_RX_BUFFER_SIZE 512
#else
#define SERIAL_RX_BUFFER_SIZE 64
#endif
#endif
#endif

The downside of this is you will probably lose your mods when you next upgrade the IDE, and they won't move with your sketch code if you take it to another platform. I would like to be able to include these defines in my sketch, but I'm not sure how to get them snuck in at the right point in the build process.

Also, this file change won't translate to other processors that install their own code under the IDE, like the Teensy family. Paul Stoffregen has kept all his defines in the individual c code files like teensy/avr/cores/teensy3/serial1.c, where changing these should let you increase the size as needed:

////////////////////////////////////////////////////////////////
// Tunable parameters (relatively safe to edit these numbers)
////////////////////////////////////////////////////////////////

#define TX_BUFFER_SIZE 64 // number of outgoing bytes to buffer
#define RX_BUFFER_SIZE 64 // number of incoming bytes to buffer

I saw a note somewhere recommending power of two buffer sizes for efficiency in index arithmetic, and I can't really imagine going anything other than 64, 128, 256, 512, etc. anyway. In my current application I'm expecting to get packets of around 250 bytes back on request, so I will set the RX buffer size to 512 to be sure it's big enough to catch the whole packet. The request code is short, so I won't increase the size of the TX buffer.