Wio Terminal KNN classifier: Machine Learning [with PlatformIO]

This tutorial describes how to implement a Wio Terminal KNN classifier. In more detail, it covers how to use a KNN classifier to classify objects using colors. To implement this Wio Terminal Machine Learning example, we will use a color sensor (TCS3200). This project derives from the ESP32 Machine Learning KNN classifier where we used the KNN classifier to recognize balls with different colors. In this simple, Wio Terminal Machine Learning tutorial, we will replace the ESP32 with Wio Terminal and we will add a color sensor because the Wio doesn’t have a built-in sensor. Moreover, we will use the built-in LCD display to show the result.

What is the KNN classifier

We have already covered what is KNN. We know that KNN stands for k-nearest neighbors algorithm. This algorithm belongs to the supervised machine learning algorithms. KNN tries to assign a label to an object measuring how similar this is to others seen during the training phase. We can have two different phases:

• training phase
• labeling phase where the trained algorithm assigns a label to an object

The features we consider are the three color components (Red, Green, Blue). Using KNN with Wio Terminal is very simple because there is already an Arduino library ready to use. Moreover, KNN is very simple so it can be implemented on a Wio Termina device.

How to use KNN with WIO Terminal to classify objects

Now that we know how KNN works, we can focus our attention on the implementation of the KNN on the Wio Terminal. We will use PlatformIO to create this project even if you can use another IDE.

The code we will use, derives from the Arduino KNN code. Anyway, we will modify it because the Wio Terminal is different from Arduino Nano 33 BLE Sense:

#include <Arduino.h>
#include <Arduino_KNN.h>
#include"TFT_eSPI.h"
// Define color sensor pins
#define S0 32
#define S1 36
#define S2 39
#define S3 40
#define sensorOut A8

/*
  k-NN color classification
  -------------------------
  This sketch classifies objects using a color sensor.
  First you 'teach' the Arduino by putting an example of each object close to the color sensor.
  After this the Arduino will guess the name of objects it is shown based on how similar
  the color is to the examples it has seen.
  This example uses a case of k-Nearest Neighbour (k-NN) algorithm where k=5.
  HARDWARE: Arduino Nano BLE Sense
  USAGE: Follow prompts in serial console. Move object close to the board to sample its color, then move it away.
  Works best in a well lit area with objects of different colors.
 
  This example code is in the public domain.
*/

const int INPUTS = 3; // Classifier input is color sensor data; red, green and blue levels
const int CLASSES = 3; // Number of objects we will classify (e.g. Apple, Banana, Orange)
const int EXAMPLES_PER_CLASS = 30; // Number of times user needs to show examples for each object
const int K = 5;
// Create a new KNNClassifier
KNNClassifier myKNN(INPUTS);
// Names for each class (object type)
String      label[CLASSES] = {"Orange", "Blue", "White"};
// Array to store data to pass to the KNN library
float color[INPUTS];
// Threshold for color brightness
const int THRESHOLD = 5;
// Handling Wio Terminal screen
TFT_eSPI tft;
#define Y_OFFSET 10

// Function to read Red Pulse Widths
int getRedPW() {
  // Set sensor to read Red only
  digitalWrite(S2,LOW);
  digitalWrite(S3,LOW);
  // Define integer to represent Pulse Width
  int PW;
  // Read the output Pulse Width
  PW = pulseIn(sensorOut, LOW);
  // Return the value
  return PW;
}
// Function to read Green Pulse Widths
int getGreenPW() {
  // Set sensor to read Green only
  digitalWrite(S2,HIGH);
  digitalWrite(S3,HIGH);
  // Define integer to represent Pulse Width
  int PW;
  // Read the output Pulse Width
  PW = pulseIn(sensorOut, LOW);
  // Return the value
  return PW;
}
// Function to read Blue Pulse Widths
int getBluePW() {
  // Set sensor to read Blue only
  digitalWrite(S2,LOW);
  digitalWrite(S3,HIGH);
  // Define integer to represent Pulse Width
  int PW;
  // Read the output Pulse Width
  PW = pulseIn(sensorOut, LOW);
  // Return the value
  return PW;
}
void readColor(float color[]) {
  int colorTotal = 0;
 // Variables for Color Pulse Width Measurements
 int redPW = 0;
 int greenPW = 0;
 int bluePW = 0;
 // Read Red Pulse Width
 redPW = getRedPW();
 // Delay to stabilize sensor
 delay(200);
 // Read Green Pulse Width
 greenPW = getGreenPW();
 // Delay to stabilize sensor
 delay(200);
 // Read Blue Pulse Width
 bluePW = getBluePW();
 // Delay to stabilize sensor
 delay(200);
 // Print output to Serial Monitor
 Serial.print("Red PW = ");
 Serial.print(redPW);
 Serial.print(" - Green PW = ");
 Serial.print(greenPW);
 Serial.print(" - Blue PW = ");
 Serial.println(bluePW);
 colorTotal = (redPW + greenPW + bluePW);
 // Normalise the color sample data and put it in the classifier input array
 color[0] = (float)redPW / colorTotal;
 color[1] = (float)greenPW / colorTotal;
 color[2] = (float)bluePW / colorTotal;
 // Print the red, green and blue percentage values
 Serial.print(color[0]);
 Serial.print(",");
 Serial.print(color[1]);
 Serial.print(",");
 Serial.println(color[2]);
}
void displayText(String text, int x, int y, int color1, int color2, int size) {
  tft.setTextColor(color1, color2);
  tft.setTextSize(size);
  if (x == -1) {
    x = (tft.width() - tft.textWidth(text)) / 2;
  }
  tft.drawString(text, x, y + Y_OFFSET);
}
void displayText(String text, int x, int y, int color1, int color2) {
  displayText(text, x,y, color1, color2, 2);
}
void checkButton() {
  while (digitalRead(WIO_KEY_A) != LOW) {
      delay(200);
  }
}
void setup() {
  // Wio Button A setup
  pinMode(WIO_KEY_A, INPUT_PULLUP);
  // Set S0 - S3 as outputs
  pinMode(S0, OUTPUT);
  pinMode(S1, OUTPUT);
  pinMode(S2, OUTPUT);
  pinMode(S3, OUTPUT);
  // Set Pulse Width scaling to 20%
  digitalWrite(S0,HIGH);
  digitalWrite(S1,LOW);
  // Init the LCD Display
  tft.begin();
  tft.setRotation(3);
  tft.fillScreen(TFT_BLACK);
  digitalWrite(LCD_BACKLIGHT, HIGH); 
  tft.setTextSize(3);
  tft.setTextColor(TFT_WHITE);
  tft.drawString("KNN Classifier", (tft.width() - tft.textWidth("KNN Classifier")) / 2, (tft.height() - tft.fontHeight()) / 2);
  delay(2000);
  tft.setTextSize(2);
  // Set Sensor output as input
  pinMode(sensorOut, INPUT);
  Serial.begin(9600);
  while (!Serial);

  Serial.println("TSC3200 + Arduino k-NN color classifier");
  
  // Ask user for the name of each object
  for (int currentClass = 0; 
       currentClass < CLASSES; 
       currentClass++) {
    tft.fillScreen(TFT_BLACK);    
    displayText("Press Button to acquire", -1, 1, TFT_WHITE, TFT_BLACK);
    displayText("samples of" , -1, 20, TFT_WHITE, TFT_BLACK);
    displayText(label[currentClass], -1, 55, TFT_YELLOW, TFT_BLACK, 3);
    checkButton();
    // Ask user to show examples of each object
    for (int currentExample = 0; 
         currentExample < EXAMPLES_PER_CLASS; 
         currentExample++) {
      Serial.print("Show me an example ");
      Serial.println(label[currentClass]);
      displayText("Sample: " +  String(currentExample), 10, 100, TFT_GREEN, TFT_BLACK);
      // Wait for an object then read its color
      readColor(color);
      // Add example color to the k-NN model
      myKNN.addExample(color, currentClass);
		  delay(300);
    }  
  }
}
void loop() {
  int classification;
  tft.fillScreen(TFT_BLACK);
  displayText("Show me something", -1, 15, TFT_CYAN, TFT_BLACK);
  displayText("to classify", -1, 35, TFT_CYAN, TFT_BLACK);
  displayText("Press button", -1, 55, TFT_CYAN, TFT_BLACK);
  // Wait for the object to move away again
  Serial.println("Show me the next object...");
  
  checkButton();
  Serial.println("Let me guess your object");
  // Wait for an object then read its color
  readColor(color);
  // Classify the object
  classification = myKNN.classify(color, K);
  // Print the classification
  Serial.print("You showed me ");
  Serial.println(label[classification]);
  displayText("You showed me", -1, 80, TFT_WHITE, TFT_BLACK);
  displayText(label[classification], -1, 105, TFT_RED, TFT_BLACK,3);
  checkButton();
  
}Code language: C++ (cpp)

while the platformio.ini file to use with this project is:

[env:seeed_wio_terminal]
platform = atmelsam
board = seeed_wio_terminal
framework = arduino
lib_deps =
   arduino-libraries/Arduino_KNN @ ^0.1.0

How the code works

In the beginning, we suppose we want to classify three different candles that have three colors:

String      label[CLASSES] = {"Orange", "Blue", "White"};Code language: C++ (cpp)

Moreover, there are three different features we use to classify or label these objects: Red, Blue, and Green. By now, forget for a while how to read colors using TCS3200 sensor, we will cover it later.

The code acquires 30 samples for each object we want to detect. Instead of using the proximity sensor, we will use a delay between different samples. Once the training phase is complete, in the loop method, the code start asking to detect objects that will be classified.

How to use Wio Terminal to detect colors and apply machine learning

To classify objects using Wio Terminal and KNN, as stated previously, we will use color. To do it, we will connect the Wio Terminal with the TCS3200 color sensor followig this schema:

S0 and S1 pins control the output scaling frequency according to this table:

In the setup() method we have used:

// Set Pulse Width scaling to 20%
digitalWrite(S0,HIGH);
digitalWrite(S1,LOW);Code language: C# (cs)

This color sensor uses an 8×8 diode array that acquires different color: Red, Green and Blue. Changing the S2 and S3 pin logic level, it is possible to select witch filter we want to apply or in other words which color we want to detect. The table below describes the logic levels of S2/S3:

Therefore, in the code to acquire the BLUE color component we have used:

// Function to read Blue Pulse Widths
int getBluePW() {
  // Set sensor to read Blue only
  digitalWrite(S2,LOW);
  digitalWrite(S3,HIGH);
  // Define integer to represent Pulse Width
  int PW;
  // Read the output Pulse Width
  PW = pulseIn(sensorOut, LOW);
  // Return the value
  return PW;
}Code language: C++ (cpp)

How to use Wio LCD to visualize the KNN classification result

Wio Terminal has a built-in color display that we can use to visualize the KNN classification results. Moreover, we can use the Wio Terminal LCD to guide the user during the training phase and the labeling phase.

How to setup LCD display in Wio Terminal

Before using the LCD display, we have to initialize it. Therefore, this piece of code includes the header file:

#include"TFT_eSPI.h"Code language: C++ (cpp)

moreover, in the setup() method the code initializes the display:

// Init the LCD Display
tft.begin();
tft.setRotation(3);
tft.fillScreen(TFT_BLACK);
digitalWrite(LCD_BACKLIGHT, HIGH); 
tft.setTextSize(3);
tft.setTextColor(TFT_WHITE);
tft.drawString("KNN Classifier", 
         (tft.width() - tft.textWidth("KNN Classifier")) / 2, 
         (tft.height() - tft.fontHeight()) / 2);
delay(2000);
tft.setTextSize(2);Code language: C++ (cpp)

This code sets the background as BLACK and shows for 2 seconds the string KNN Classifiier. The result is shown below:

Handling button to acquire sample

The next step, is handling the button present on top of the Wio Terminal. When user presses this button, Wio Terminal starts acquiring sample for the KNN classification. To do it, we use this code:

void checkButton() {
  while (digitalRead(WIO_KEY_A) != LOW) {
      delay(200);
  }
}Code language: C# (cs)

while in the setup() method:

// Wio Button A setup
pinMode(WIO_KEY_A, INPUT_PULLUP);Code language: PHP (php)

Finally, every time we have to show a message to the user, we use this code:

void displayText(String text, int x, int y, int color1, int color2, int size) {
  tft.setTextColor(color1, color2);
  tft.setTextSize(size);
  if (x == -1) {
    x = (tft.width() - tft.textWidth(text)) / 2;
  }
  tft.drawString(text, x, y + Y_OFFSET);
}
void displayText(String text, int x, int y, int color1, int color2) {
  displayText(text, x,y, color1, color2, 2);
}Code language: C# (cs)

More resources:
How to use Tensorflow JS with ESP32-CAM
How to use Tensorflow with ESP32
ESP32 KNN classification
Wio Terminal MQTT client
How to create a Weather station with Wio Terminal
How to use Tensorflow micro with ESP32

How to test the classification with KNN and Wio Terminal

To test if the classification works, we need three diferent objects with different colors. I used three candles with orange, blue and white color. Let’s run the code and during training phase, you should see somthing like the picture shown below:

The same thing happens for the other colors. Once the training phase is complete, we can start testing the classification using the Wio Terminal KNN and the TCS3200. You can point the TCS3200 to one of the objects.

Below the results:

You can try by yourself and check that it works!

Wrapping up

At the end of this post, we have covered how to use the Wio Terminal KNN classifier to classify objects using colors. We have integrated the Wio Terminal with TCS3200 to detect color and we have used the Arduino KNN library. Starting from the Arduino source code developed for Nano 33 BLE Sense, we have modified this code to adapt it to the Wio Terminal and TCS3200 so that we can use KNN. Moreover, we have discovered how to use a simple Machine Learning algorithm with the Wio Terminal.