Integrate ESP8266 and Android using Android speech recognition

This ESP8266 Programming tutorial describes how to use integrate ESP8266 and Android using voice commands. In other words, in this article, we will use our voice to send a set of commands to an IoT device. During this post, we will build a voice-activated IoT project using ESP8266 and Android. This is an interesting topic because this project uses a different way to interact with IoT device. Usually, we are used to interacting with a device using a simple user interface exposed by the device or a using a smartphone app that sends commands to the device.

How to use ESP8266 with Android

The idea that stands behind this ESP8266 Programming tutorial is the integration between ESP8266 and Android using the voice commands. Using such commands it is possible to control a device like Arduino or ESP8266. To build this voice-activated project, we develop an Android app that is the mean that captures the user’s voice and transforms it into a set of commands that are sent to the ESP8266. The picture below describes the project overview:

In more details this project is made by two different sub-systems:

• Android app
• IoT app

The Android app takes care to interact with the user and listens to the voice commands. Next, the app translates the voice commands into commands that the IoT device can understand. In this article, as IoT device, we will use an ESP8266 WeMos that controls a NeoPixel Ring. You can use an Arduino Uno instead of the ESP or a MKR1000. There are other several projects that we can build using ESP8266, you can discover how to send IoT notification using ESP8266.

How to develop a speech recognition Android app to connect ESP8266 and Android

The first step in this project is developing an Android app that recognizes user speech. Fortunately, Android provides a built-in system that is able to recognize the user words. The app user interface is very simple. There is only one button that we use to start sending commands:

The layout is very simple as shown below:

[xml] <android.support.constraint.ConstraintLayout
xmlns:android=’http://schemas.android.com/apk/res/android’
xmlns:app=’http://schemas.android.com/apk/res-auto’
xmlns:tools=’http://schemas.android.com/tools’
android:layout_width=’match_parent’
android:layout_height=’match_parent’
tools:context=ìcom.survivingwithandroid.voice.MainActivity’
android:id=’@+id/mainView’
android:background=’@drawable/bg_gradient’>
<Button android:layout_width=’wrap_content’
android:layout_height=’wrap_content’
app:layout_constraintLeft_toRightOf=’parent’
app:layout_constraintRight_toLeftOf=’parent’
android:text=’Send command’
android:id=’@+id/btnCommand’
app:layout_constraintBottom_toBottomOf=’parent’
android:layout_marginBottom=’15dp’/>
</android.support.constraint.ConstraintLayout>
[/xml]

The next step is overriding in the MainActivity.java the onCreate method:

[java] @Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
btn = (Button) findViewById(R.id.btnCommand);
btn.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View v) {
startVoiceCommand();
}
}[/java]

where the startVoiceCommand() is the method that handles the voice interaction with the user. In this context, to capture the user’s voice the app uses an Intent delegating all the hard work to the Android OS:

[java] private void startVoiceCommand() {
Log.d(TAG, &quot;Starting Voice intent…&quot;);
Intent i = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
RecognizerIntent.LANGUAGE_MODE);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());
i.putExtra(RecognizerIntent.EXTRA_PROMPT, "Tell me, I’m ready!");
try {
startActivityForResult(i, REQ_SPEECH_RESULT);
}
catch (Exception e) {
Snackbar.make(v, "Speech to text not supported",
Snackbar.LENGTH_LONG).show();
}
} [/java]

This method is very simple, it invokes the intent RecognizerIntent.ACTION_RECOGNIZE_SPEECH providing some configuration parameters as the current locale and the message we want to show to the user. When the user clicks on the button the app shows the dialog waiting for the voice input. Finally, the app starts the intent waiting for the result:

To this purpose, the app overrides the method onActivityResult:

[java] @Override protected void onActivityResult(int requestCode,
int resultCode,
Intent data) {
super.onActivityResult(requestCode, resultCode, data);
// Check the Request code
if (requestCode == REQ_SPEECH_RESULT) {
Log.d(TAG, "Request speech result..");
ArrayList<String> results =
data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);
String command = results.get(0);
Log.d(TAG, "Current command ["+command+"]");
// Now we send commands to the IoT device
}
} [/java]

In the method above, the app extracts the command and according to it invoke the ESP8266 to set the ring LEDs color. In this example, the Android IoT voice app handles simple commands like Red, Green, Blue and so on.

If you are new to Android and want to have more information you can read this tutorial describing how to develop an Android app using material design.

Exchanging data between ESP8266 and Android app

In this step of this ESP8266 programming project, we implement the network communication between Android and ESP8266. By now, we can suppose that the ESP8266 exposes a method invoked by the Android app to set the ring colors. In this context, we can suppose that the ESP8266 exposes a RESTful API. In order to invoke this API, the Android app uses an HTTP connection.

To this purpose, it is necessary to create a new class called IoTConnectionHandler that handles all the network details. The class is shown below:

[java] public class IoTConnectionHandler {
private static IoTConnectionHandler me;
private OkHttpClient client;
private static final String TAG = IoTConnectionHandler.class.getName();
private static final String IOT_URL ="http://192.168.1.9:8080/ring?param=0";
private IoTConnectionHandler() {
client = new OkHttpClient();
}
public static IoTConnectionHandler getInstance() {
if (me == null)
me = new IoTConnectionHandler();
return me;
}
public void sendData(String data) {
Request req = new Request.Builder()
.url(IOT_URL + data)
.build();
client.newCall(req).enqueue(new Callback() {
@Override
public void onFailure(Call call, IOException e) {
Log.e(TAG, "Connection error", e);
}
@Override
public void onResponse(Call call, Response response) throws IOException {
Log.i(TAG, "Command sent");
}
});
}
} [/java]

It is very simple and it uses OkHTTP library. Notice that the data parameter holds the color hex code retrieved from the voice command. The next part is implementing the IoT side of the project that receives the color hex code using the API exposed by the device and set the LEDs color.

Integrating ESP8266 and Android

In this step of this voice-controlled IoT device, we will develop the code necessary to:

• expose an API in ESP8266 invoked by the Android app
• control a Neopixel RGB ring

Before diving into the IoT project details, it is useful to know how to connect the ESP8266 to the Neopixel RGB ring:

To simplify the code development the IoT devices uses these two libraries:

• Adafruit Neopixel library
• aRest library

The first one is used to control the LEDs ring while the second library is necessary to expose some functions in the sketch as API. If you are new to this library, you can read my previous post describing how to expose Arduino functions as API.

The code below is the sketch:

[c] #include <Adafruit_NeoPixel.h>
#include <ESP8266WiFi.h>
#include <aREST.h>
#define PIN D2
#define NUMS 12
#define SERVER_PORT 8080
// Neopixel rings
Adafruit_NeoPixel pixels = Adafruit_NeoPixel(12, PIN, NEO_GRB + NEO_KHZ800);
aREST rest = aREST();
char *ssid = "xxxxx"
char *pwd = "xxx"
// Let us create the server
WiFiServer server(SERVER_PORT);
void setup() {
Serial.begin(9600);
pixels.begin();
pixels.setBrightness(85);
// Register the function
rest.function("ring", setColor);
WiFi.begin(ssid, pwd);
Serial.println("Connecting to WiFi…");
while (WiFi.status() != WL_CONNECTED) {
delay(1000);
Serial.println("Try again….");
}
Serial.println("WiFi connected…");
// let us start the server server.begin();
}
void loop() {
WiFiClient client = server.available();
if (!client) { return ; }
while (!client.available()) { delay(1); }
rest.handle(client);
}

int setColor(String color) {
Serial.println(&quot;Hex color [‘ + color + ‘]’);
long tmpColor = strtol( ‘(‘#’ + color)[1], NULL, 16);
Serial.println("Int ["+String(tmpColor)+"]");
int r = tmpColor << 16;
int g = tmpColor << 8 && 0xFF;
int b = tmpColor & 0xFF;
Serial.print(‘Red [‘ + String(r) +’]’);
Serial.print(‘Green [‘ + String(g) + ‘]’);
Serial.println(‘Blue [‘ + String(b) +’]’);
for (int i = 0; i < 12; i++)
pixels.setPixelColor(i, pixels.Color(r,g,b));
pixels.show();
return 1; } [/c]

In details, at the beginning, the sketch tries to connect to the WiFi network. To this purpose, you have to provide the WiFi ssid and the password. Once the connection is established, the sketch configures the server and its port. Moreover, it declares the function we want to export as API.

The last part is the method used to control the Neopixel ring. You could be interested on how to use ESP8266 with Telegram to automate some tasks and controlling LEDs.

As soon as we send a voice command to the IoT device app shows these logs:

Summary

At the end of this ESP8266 Programming tutorial, you know how to use voice to control an IoT device integrating ESP8266 and Android app. In more details, you have explored how to connect and Android app with an ESP8266 and how to use voice to interact with it.