ESP32 Machine Learning: ESP32 KNN classifier

This tutorial describes how to use ESP32 Machine Learning. In more detail, it covers how to use an ESP32 KNN classifier to classify objects using their colors. To implement this ESP32 Machine Learning example, we will use a color sensor (TCS3200). This project derives from the Arduino Blog where it was used a KNN classifier to recognize different fruits. In this simple ESP32 KNN Machine Learning tutorial, we will replace the Arduino Nano 33 BLE with the ESP32 and we will add a color sensor because the ESP32 doesn’t have a built-in sensor. We will apply the ESP32 KNN classifier to simple balls.

What is the KNN classifier

To better understand the project we are going to build, it is useful to have an idea about how the classifier works. Briefly, KNN stands for k-nearest neighbors algorithm. This algorithm belongs to the supervises machine learning algorithms. KNN assumes that similar objects are near to each other. The KNN tries to assign a label to an object measuring how similar this is to others seen during the training phase.

Therefore, in this ESP32 KNN Machine Learning example we have two different phases:

  • first phase where we train the alogorithm so that it can recognizes similaraties
  • we apply the algorithm to object to identify it

The features we consider are the three color components (Red, Green, Blue). Using KNN with ESP32 is very simple because there is already an Arduino library ready to use. Moreover, KNN is very simple so it can be implemented on an ESP32 device. In this way, with a few lines of code, we can try how to use Machine Learning with ESP32.

How to use KNN with ESP32 to classify objects

Now that we know how KNN works, we can focus our attention on the implementation of the KNN on the ESP32. We will use PlatformIO to create this project even if you can use another IDE.

First of all, create a new project. The code that we will use here uses the Arduino code in github modifying it in different places to support the TCS3200 color sensor. The code is shown below:

#include <Arduino.h> #include <Arduino_KNN.h> // Define color sensor pins #define S0 23 #define S1 22 #define S2 19 #define S3 21 #define sensorOut 18 /* k-NN color classification Arduino original version at https://github.com/arduino-libraries/Arduino_KNN/blob/master/examples/ColorClassifier/ColorClassifier.ino */ const int INPUTS = 3; // Classifier input is color sensor data; red, green and blue levels const int CLASSES = 3; // Number of objects we will classify (e.g. Apple, Banana, Orange) const int EXAMPLES_PER_CLASS = 30; // Number of times user needs to show examples for each object const int K = 5; // Create a new KNNClassifier KNNClassifier myKNN(INPUTS); // Names for each class (object type) // These are the color of the balls String label[CLASSES] = {"Silver", "Blue", "Red"}; // Array to store data to pass to the KNN library float color[INPUTS]; // Threshold for color brightness const int THRESHOLD = 5; // Function to read Red Pulse Widths int getRedPW() { // Set sensor to read Red only digitalWrite(S2,LOW); digitalWrite(S3,LOW); // Define integer to represent Pulse Width int PW; // Read the output Pulse Width PW = pulseIn(sensorOut, LOW); // Return the value return PW; } // Function to read Green component int getGreenPW() { digitalWrite(S2,HIGH); digitalWrite(S3,HIGH); // Define integer to represent Pulse Width int PW; PW = pulseIn(sensorOut, LOW); return PW; } // Function to read Blue component int getBluePW() { digitalWrite(S2,LOW); digitalWrite(S3,HIGH); int PW; PW = pulseIn(sensorOut, LOW); return PW; } void readColor(float color[]) { int colorTotal = 0; int redPW = 0; int greenPW = 0; int bluePW = 0; // Read Red Pulse Width redPW = getRedPW(); // Delay to stabilize sensor delay(200); // Read Green Pulse Width greenPW = getGreenPW(); // Delay to stabilize sensor delay(200); // Read Blue Pulse Width bluePW = getBluePW(); // Delay to stabilize sensor delay(200); // Print output to Serial Monitor Serial.print("Red PW = "); Serial.print(redPW); Serial.print(" - Green PW = "); Serial.print(greenPW); Serial.print(" - Blue PW = "); Serial.println(bluePW); colorTotal = (redPW + greenPW + bluePW); // Normalise the color sample data and put it in // the classifier input array color[0] = (float)redPW / colorTotal; color[1] = (float)greenPW / colorTotal; color[2] = (float)bluePW / colorTotal; // Print the red, green and blue percentage values Serial.print(color[0]); Serial.print(","); Serial.print(color[1]); Serial.print(","); Serial.println(color[2]); } void setup() { // Set S0 - S3 as outputs pinMode(S0, OUTPUT); pinMode(S1, OUTPUT); pinMode(S2, OUTPUT); pinMode(S3, OUTPUT); // Set Pulse Width scaling to 20% digitalWrite(S0,HIGH); digitalWrite(S1,LOW); // Set Sensor output as input pinMode(sensorOut, INPUT); Serial.begin(9600); while (!Serial); Serial.println("TSC3200 + ESP32 k-NN color classifier"); delay(5000); // Ask user for the name of each object for (int currentClass = 0; currentClass < CLASSES; currentClass++) { // Ask user to show examples of each object for (int currentExample = 0; currentExample < EXAMPLES_PER_CLASS; currentExample++) { Serial.print("Show me an example "); Serial.println(label[currentClass]); // Wait for an object then read its color readColor(color); // Add example color to the k-NN model myKNN.addExample(color, currentClass); delay(500); } Serial.println("Wait for 5 sec. before acquiring next sample"); delay(5000); } } void loop() { int classification; // Wait for the object to move away again Serial.println("Show me the next object..."); delay(2000); Serial.println("Let me guess your object"); // Wait for an object then read its color readColor(color); // Classify the object classification = myKNN.classify(color, K); // Print the classification Serial.print("You showed me "); Serial.println(label[classification]); }
Code language: C++ (cpp)

while the platformio.ini file to use with this project is:

[env:esp32dev] platform = espressif32 board = esp32dev framework = arduino upload_port=/dev/tty.SLAB_USBtoUART lib_deps = arduino-libraries/Arduino_KNN @ ^0.1.0 adafruit/Adafruit NeoPixel @ ^1.7.0 #optional
Code language: PHP (php)

More useful resources:

How the ESP32 KNN classifier code works

In the beginning, we suppose we want to classify three different balls that have three colors:

String label[CLASSES] = {"Silver", "Blue", "Red"};
Code language: C++ (cpp)

Moreover, there are three different features we use to classify or label these objects: Red, Blue, and Green. By now, forget for a while how to read colors using TCS3200 sensor, we will cover it later.

The code acquires 30 samples for each object we want to detect. Instead of using the proximity sensor, we will use a delay between different samples. Once the training phase is complete, in the loop method, the code start asking to detect objects that will be classified.

How to use ESP32 to detect colors and apply machine learning

To classify object using ESP32 and KNN, as stated previously, we will use color. To do it, we will connect the ESP32 with TCS3200 color sensor followig this schema:

ESP32 PinTCS3200 Pin
VccVcc
GNDGND
23S0
22S1
19S2
21S3
18Out

S0 and S1 pins control the output scaling frequency according to this table:

S0 PinS1 PinFrequency
LowLowPower down
LowHigh2%
HighLow20%
HighHigh100%

In the setup() method we have used:

// Set Pulse Width scaling to 20% digitalWrite(S0,HIGH); digitalWrite(S1,LOW);
Code language: C# (cs)

This color sensor uses an 8×8 diode array that acquires different colors: Red, Green, and Blue. Changing the S2 and S3 pin logic level, it is possible to select which filter we want to apply or in other words which color we want to detect. The table below describes the logic levels of S2/S3:

S2 PinS3 PinPhotodiode type
LowLowRED
LowHighBLUE
HighLowNo filter
HighHighGREEN

Therefore, in the code to acquire the BLUE color component we have used:

// Function to read Blue Pulse Widths int getBluePW() { // Set sensor to read Blue only digitalWrite(S2,LOW); digitalWrite(S3,HIGH); // Define integer to represent Pulse Width int PW; // Read the output Pulse Width PW = pulseIn(sensorOut, LOW); // Return the value return PW; }
Code language: C++ (cpp)

How to visualize the ESP32 KNN classification result

This part is optional. We want to add a simple visualization feature using Neopixel 4×4 LEDs. We use it to visualize the result of the classification provided by the KNN with the ESP32.

To add this feature, you have to add the following line in:

#include <Adafruit_NeoPixel.h>
Code language: C++ (cpp)

Then, we have to define the Neopixel handler:

// Neopixel visual display Adafruit_NeoPixel strip = Adafruit_NeoPixel(16, 5, NEO_GRB + NEO_KHZ800); int leds[] = {0,1,1,0,1,1,1,1,1,1,1,1,0,1,1,0};
Code language: C++ (cpp)

The leds array contains the list of the array to turn on. Next, in the setup() method:

strip.begin(); strip.setBrightness(255); strip.fill(0,0,0); strip.show();
Code language: C++ (cpp)

In this way, we turn off all the LEDs. Finally, to visualize the classification result we use this code:

uint32_t color = 0; switch(classification) { case 0: color = strip.Color(255,255,255); break; case 1: color = strip.Color(0,0,255); break; case 2: color = strip.Color(255,0,0); } // show leds color for (int i=0; i < 16; i++) { if (leds[i] == 1) strip.setPixelColor(i, color); } strip.show();
Code language: JavaScript (javascript)

That’s all!

If you want to try more ESP32 projects read:
How to use Tensorflow JS with ESP32-CAM
How to use Tensorflow with ESP32
ESP32 Image classification

Schematic

The schematic is shown below:

ESP32 KNN schematic to classify colors using TCS3200

How to test the classification with KNN and ESP32

To test if the classification works, we need three diferent objects with different colors. I used three christmass ball, but you can use other kinds of objects. Let’s run the code and during training phase in the log you should see somthing like the picture shown below:

ESP32 KNN training

and for blue:

The same thing happens for the other object. Once the training phase is complete, we can start test the classification using the ESP32 KNN and the TCS3200.

Below the results:

If you prefer to use the output console, the result is shown below:

As you can see it works!

Wrapping up

At the end of this post, we have covered how to use the ESP32 KNN classifier to classify objects using colors. We have integrated the ESP32 with TCS3200 to detect color and we have used the Arduino KNN library. Starting from the Arduino source code developed for Nano 33 BLE Sense, we have modified this code to adapt it to the ESP32 and TCS3200 so that we can use KNN. Moreover, we have discovered how to use a simple Machine Learning algorithm with the ESP32.

    1. Artem January 10, 2021
      • Francesco Azzola January 11, 2021

    Add Your Comment