ESP32 Machine Learning: ESP32 KNN classifier

This tutorial describes how to use ESP32 Machine Learning. In more detail, it covers how to use an ESP32 KNN classifier to classify objects using their colors. To implement this ESP32 Machine Learning example, we will use a color sensor (TCS3200). This project derives from the Arduino Blog where it was used a KNN classifier to recognize different fruits. In this simple ESP32 KNN Machine Learning tutorial, we will replace the Arduino Nano 33 BLE with the ESP32 and we will add a color sensor because the ESP32 doesn’t have a built-in sensor. We will apply the ESP32 KNN classifier to simple balls.

What is the KNN classifier

To better understand the project we are going to build, it is useful to have an idea about how the classifier works. Briefly, KNN stands for k-nearest neighbors algorithm. This algorithm belongs to the supervises machine learning algorithms. KNN assumes that similar objects are near to each other. The KNN tries to assign a label to an object measuring how similar this is to others seen during the training phase.

Therefore, in this ESP32 KNN Machine Learning example we have two different phases:

• first phase where we train the alogorithm so that it can recognizes similaraties
• we apply the algorithm to object to identify it

The features we consider are the three color components (Red, Green, Blue). Using KNN with ESP32 is very simple because there is already an Arduino library ready to use. Moreover, KNN is very simple so it can be implemented on an ESP32 device. In this way, with a few lines of code, we can try how to use Machine Learning with ESP32.

How to use KNN with ESP32 to classify objects

Now that we know how KNN works, we can focus our attention on the implementation of the KNN on the ESP32. We will use PlatformIO to create this project even if you can use another IDE.

First of all, create a new project. The code that we will use here uses the Arduino code in github modifying it in different places to support the TCS3200 color sensor. The code is shown below:

#include <Arduino.h>
#include <Arduino_KNN.h>
// Define color sensor pins
#define S0 23
#define S1 22
#define S2 19
#define S3 21
#define sensorOut 18

/*
  k-NN color classification
  Arduino original version at 
  https://github.com/arduino-libraries/Arduino_KNN/blob/master/examples/ColorClassifier/ColorClassifier.ino
*/
const int INPUTS = 3; // Classifier input is color sensor data; red, green and blue levels
const int CLASSES = 3; // Number of objects we will classify (e.g. Apple, Banana, Orange)
const int EXAMPLES_PER_CLASS = 30; // Number of times user needs to show examples for each object
const int K = 5;
// Create a new KNNClassifier
KNNClassifier myKNN(INPUTS);
// Names for each class (object type)
// These are the color of the balls
String      label[CLASSES] = {"Silver", "Blue", "Red"};
// Array to store data to pass to the KNN library
float color[INPUTS];
// Threshold for color brightness
const int THRESHOLD = 5;
// Function to read Red Pulse Widths
int getRedPW() {
	// Set sensor to read Red only
	digitalWrite(S2,LOW);
	digitalWrite(S3,LOW);
	// Define integer to represent Pulse Width
	int PW;
	// Read the output Pulse Width
	PW = pulseIn(sensorOut, LOW);
	// Return the value
	return PW;
}
// Function to read Green component
int getGreenPW() {
	digitalWrite(S2,HIGH);
	digitalWrite(S3,HIGH);
	// Define integer to represent Pulse Width
	int PW;
	PW = pulseIn(sensorOut, LOW);
	return PW;
}
// Function to read Blue component
int getBluePW() {
	digitalWrite(S2,LOW);
	digitalWrite(S3,HIGH);
	int PW;
	PW = pulseIn(sensorOut, LOW);
	return PW;
}
void readColor(float color[]) {
  int colorTotal = 0;
  int redPW = 0;
  int greenPW = 0;
  int bluePW = 0;  // Read Red Pulse Width
  redPW = getRedPW();
  // Delay to stabilize sensor
  delay(200);
  // Read Green Pulse Width
  greenPW = getGreenPW();
  // Delay to stabilize sensor
  delay(200);
  // Read Blue Pulse Width
  bluePW = getBluePW();
  // Delay to stabilize sensor
  delay(200);
  // Print output to Serial Monitor
  Serial.print("Red PW = ");
  Serial.print(redPW);
  Serial.print(" - Green PW = ");
  Serial.print(greenPW);
  Serial.print(" - Blue PW = ");
  Serial.println(bluePW);
  colorTotal = (redPW + greenPW + bluePW);
  // Normalise the color sample data and put it in 
  // the classifier input array
  color[0] = (float)redPW / colorTotal;
  color[1] = (float)greenPW / colorTotal;
  color[2] = (float)bluePW / colorTotal;
  // Print the red, green and blue percentage values
  Serial.print(color[0]);
  Serial.print(",");
  Serial.print(color[1]);
  Serial.print(",");
  Serial.println(color[2]);
}
void setup() {
  // Set S0 - S3 as outputs
  pinMode(S0, OUTPUT);
  pinMode(S1, OUTPUT);
  pinMode(S2, OUTPUT);
  pinMode(S3, OUTPUT);
  // Set Pulse Width scaling to 20%
  digitalWrite(S0,HIGH);
  digitalWrite(S1,LOW);
  // Set Sensor output as input
  pinMode(sensorOut, INPUT);
  Serial.begin(9600);
  while (!Serial);

  Serial.println("TSC3200 + ESP32 k-NN color classifier");
  delay(5000);
  // Ask user for the name of each object
  for (int currentClass = 0; currentClass < CLASSES; currentClass++) {
  
    // Ask user to show examples of each object
    for (int currentExample = 0; currentExample < EXAMPLES_PER_CLASS; currentExample++) {
      Serial.print("Show me an example ");
      Serial.println(label[currentClass]);
      // Wait for an object then read its color
      readColor(color);
      // Add example color to the k-NN model
      myKNN.addExample(color, currentClass);
      delay(500);
    }
    
    Serial.println("Wait for 5 sec. before acquiring next sample");
    delay(5000);
  }
}
void loop() {
  int classification;
  // Wait for the object to move away again
  Serial.println("Show me the next object...");
  delay(2000);
  Serial.println("Let me guess your object");
  // Wait for an object then read its color
  readColor(color);
  // Classify the object
  classification = myKNN.classify(color, K);
  // Print the classification
  Serial.print("You showed me ");
  Serial.println(label[classification]);
 
}
Code language: C++ (cpp)

while the platformio.ini file to use with this project is:

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino
upload_port=/dev/tty.SLAB_USBtoUART
lib_deps =
   arduino-libraries/Arduino_KNN @ ^0.1.0
   adafruit/Adafruit NeoPixel @ ^1.7.0 #optionalCode language: PHP (php)

More useful resources:

• Wio Terminal KNN classifier: Machine Learning [with PlatformIO]
• How to connect Android app to Arduino using HTTP and Arduino Webserver
• Raspberry Pi Tensorflow Lite: Image classification and Object detection – Easy guide
• TinyML ESP32-CAM: Edge Image classification with Edge Impulse
• ESP32 Rest API: How to implement an ESP32 API server using PlatformIO

How the ESP32 KNN classifier code works

In the beginning, we suppose we want to classify three different balls that have three colors:

String      label[CLASSES] = {"Silver", "Blue", "Red"};Code language: C++ (cpp)

Moreover, there are three different features we use to classify or label these objects: Red, Blue, and Green. By now, forget for a while how to read colors using TCS3200 sensor, we will cover it later.

The code acquires 30 samples for each object we want to detect. Instead of using the proximity sensor, we will use a delay between different samples. Once the training phase is complete, in the loop method, the code start asking to detect objects that will be classified.

How to use ESP32 to detect colors and apply machine learning

To classify object using ESP32 and KNN, as stated previously, we will use color. To do it, we will connect the ESP32 with TCS3200 color sensor followig this schema:

S0 and S1 pins control the output scaling frequency according to this table:

In the setup() method we have used:

// Set Pulse Width scaling to 20%
digitalWrite(S0,HIGH);
digitalWrite(S1,LOW);Code language: C# (cs)

This color sensor uses an 8×8 diode array that acquires different colors: Red, Green, and Blue. Changing the S2 and S3 pin logic level, it is possible to select which filter we want to apply or in other words which color we want to detect. The table below describes the logic levels of S2/S3:

Therefore, in the code to acquire the BLUE color component we have used:

// Function to read Blue Pulse Widths
int getBluePW() {
  // Set sensor to read Blue only
  digitalWrite(S2,LOW);
  digitalWrite(S3,HIGH);
  // Define integer to represent Pulse Width
  int PW;
  // Read the output Pulse Width
  PW = pulseIn(sensorOut, LOW);
  // Return the value
  return PW;
}Code language: C++ (cpp)

How to visualize the ESP32 KNN classification result

This part is optional. We want to add a simple visualization feature using Neopixel 4×4 LEDs. We use it to visualize the result of the classification provided by the KNN with the ESP32.

To add this feature, you have to add the following line in:

#include <Adafruit_NeoPixel.h>Code language: C++ (cpp)

Then, we have to define the Neopixel handler:

// Neopixel visual display
Adafruit_NeoPixel strip = Adafruit_NeoPixel(16, 5, 
                          NEO_GRB + NEO_KHZ800);
int leds[] = {0,1,1,0,1,1,1,1,1,1,1,1,0,1,1,0};Code language: C++ (cpp)

The leds array contains the list of the array to turn on. Next, in the setup() method:

strip.begin();
strip.setBrightness(255);
strip.fill(0,0,0);
strip.show();Code language: C++ (cpp)

In this way, we turn off all the LEDs. Finally, to visualize the classification result we use this code:

uint32_t color = 0;
switch(classification) {
  case 0:
    color = strip.Color(255,255,255);
    break;
  case 1:
    color = strip.Color(0,0,255);
    break;
  case 2:
    color = strip.Color(255,0,0);
  }
  // show leds color
  for (int i=0; i < 16; i++) {
    if (leds[i] == 1)
      strip.setPixelColor(i, color);
  }
  strip.show();Code language: JavaScript (javascript)

That’s all!

If you want to try more ESP32 projects read:
How to use Tensorflow JS with ESP32-CAM
How to use Tensorflow with ESP32
ESP32 Image classification

Schematic

The schematic is shown below:

How to test the classification with KNN and ESP32

To test if the classification works, we need three diferent objects with different colors. I used three christmass ball, but you can use other kinds of objects. Let’s run the code and during training phase in the log you should see somthing like the picture shown below:

and for blue:

The same thing happens for the other object. Once the training phase is complete, we can start test the classification using the ESP32 KNN and the TCS3200.

Below the results:

If you prefer to use the output console, the result is shown below:

As you can see it works!

Wrapping up

At the end of this post, we have covered how to use the ESP32 KNN classifier to classify objects using colors. We have integrated the ESP32 with TCS3200 to detect color and we have used the Arduino KNN library. Starting from the Arduino source code developed for Nano 33 BLE Sense, we have modified this code to adapt it to the ESP32 and TCS3200 so that we can use KNN. Moreover, we have discovered how to use a simple Machine Learning algorithm with the ESP32.