This article describes how to develop a Machine Learning Android app using Fritz.ai. Before diving into the details about how to build a Machine learning Android app, it is useful to describe briefly what is Fritz.ai platform. As you may know, Machine Learning is an interesting topic that is gaining importance and promises to transform several areas including the way we interact with Android apps.
To experiment how to apply Machine Learning to Android using Fritz.ai we will develop an Android app that uses Image Classification.
Machine Learning is an application of AI that gives to a system the capability to accomplish tasks without using explicit instructions but learning from the data and improving from experience.
You can download the source code here:
What is a Fritz.ai Machine Learning platform?
Fritz.ai is a platform for Android and iOS that simplifies the development of an Android Machine Learning app. When developing an Android app, usually it is necessary to create an ML-model that will be used by the Android app. The process to build an ML-model is tedious and difficult. Fritz.ai provides a set of built-in models ready to use that speeds up the development of a Machine Learning Android app.
The built-in ML-models provided by Fritz.ai are:
- Image Labeling
- Object detection
- Style transfer
- Image segmentation
- Pose estimation
Moreover, it is possible to upload a custom model if we want to use a specific model.
How to configure Fritz.ai
The first step to use Fritz.ai Machine Learning platform is creating a free account and then create a new Machine Learning Android project, as shown in the picture below:
Now, it is possible to add a new project:
You have to provide the Project name. Once the project is created, it is necessary to provide the project package:
How to set up an Android app using Fritz.ai Image classification
Once the project is correctly configured in Fritz.ai console, we can set up the Android project to include Fritz.ai. At the project-level, it is necessary to modify build.gradle in this way:
...
allprojects {
repositories {
google()
jcenter()
maven { url "https://raw.github.com/fritzlabs/fritz-repository/master" }
}
}
...and in the build.gradle at app-level:
dependencies {
....
implementation 'ai.fritz:core:3.0.1'
implementation 'ai.fritz:vision:3.0.1'
}The last step is registering a service so that the Android app can receive model updates. To do it in the Manifest.xml it is necessary to add:
<service android:name="ai.fritz.core.FritzCustomModelService" android:exported="true" android:permission="android.permission.BIND_JOB_SERVICE" />
That’s all. We are now ready to start developing our image classification Android app.
Implementing Machine Learning image classification in Android
To implement an Android app that uses image classification we have to follow these steps:
- Configure Fritz.ai SDK in the Android app
- Label the image detected
Configure Fritz.ai SDK in Android app
The first step is configuring the Fritz.ai SDK in our Android app so that we can use it later when we need to label the captured image. Let us create an activity and in onCreate() method add the following lines:
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
Fritz.configure(this, "your_fritz.ai_key");
downloadModel();
}In line 4, it is necessary to configure Fritz using the API_KEY we get during the project set up in the console. The next step, it is downloading the model from the cloud:
private void downloadModel() {
FritzManagedModel model = new ImageLabelManagedModel();
Log.d(TAG, "Download model...");
FritzVision.ImageLabeling.loadPredictor(model,
new PredictorStatusListener<FritzVisionLabelPredictor>() {
@Override
public void onPredictorReady(FritzVisionLabelPredictor predictor) {
Log.d(TAG, "Model downloaded");
MainActivity.this.predictor = predictor;
}
});
}
In this case, the Android app uses the ImageLabelManageModel because we want to label the image using Machine Learning.
<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
<TextureView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_alignParentTop="true"
android:id="@+id/preview"/>
<Button
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:id="@+id/btnLabel"
android:text="Label it"
android:layout_alignParentBottom="true"/>
</RelativeLayout>The TextureView will hold the image stream coming from the camera, while the button will be used to label the image, as we will see later.
How to use Fritz.ai to classify an image
To classify the image, once we have an image from the camera, it is necessary to create an instance of FritzVisionImage. Let us suppose, by now, that we have an instance of ImageReader that holds the image captured from the camera. It is necessary to convert it into a bitmap and then use the predictor to label the image:
Image image = reader.acquireLatestImage(); ByteBuffer buffer = image.getPlanes()[0].getBuffer(); byte[] bytes = new byte[buffer.capacity()]; buffer.get(bytes); Bitmap bitmapImage = BitmapFactory.decodeByteArray(bytes, 0, bytes.length, null); FritzVisionImage fvi = FritzVisionImage.fromBitmap(bitmapImage);
Finally, at line 8, the predictor, that will classify the image, is ready to be used. Now, the Android app can retrieve a list of labels with relative accuracy:
FritzVisionLabelResult labels = predictor.predict(fvi);
List<FritzVisionLabel> labelList = labels.getVisionLabels();
if (labelList != null && labelList.size() > 0) {
FritzVisionLabel label = labels.getVisionLabels().get(0);
System.out.println("Label [" + label.getText() + "]");
}The app gets the first element of the list, that has higher accuracy. Now we can describe how to use the camera in Android to capture an image.
Handle the Camera in Android
Once the Machine Learning engine is ready, we have to focus our attention on how to capture the image. As stated before, the TextureView will hold the image stream coming from the camera. It is necessary to add a listener to the TextureView to know when we can use the camera:
TextureView.SurfaceTextureListener surfaceListener = new TextureView.SurfaceTextureListener() {
@Override
public void onSurfaceTextureAvailable(SurfaceTexture surface, int width, int height) {
Log.d(TAG, "Surface available");
openCamera();
}
@Override
public void onSurfaceTextureSizeChanged(SurfaceTexture surface, int width, int height) {}
@Override
public boolean onSurfaceTextureDestroyed(SurfaceTexture surface) {
return false;
}
@Override
public void onSurfaceTextureUpdated(SurfaceTexture surface) {
}
};When the surface is ready, the app can open the camera and start using it. Please notice that:
preview = (TextureView) findViewById(R.id.preview); preview.setSurfaceTextureListener(surfaceListener);
Now it is possible to open the camera:
private void openCamera() {
CameraManager manager = (CameraManager)
getSystemService(Context.CAMERA_SERVICE);
// We get the first available camera
try {
cameraId = manager.getCameraIdList()[0];
CameraCharacteristics characteristics =
manager.getCameraCharacteristics(cameraId);
StreamConfigurationMap streamConfMap = characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
imageDimension = streamConfMap.getOutputSizes(SurfaceTexture.class)[0];
// We can open the camera now
manager.openCamera(cameraId, new cameraDevice.StateCallback() {
@Override
public void onOpened(@NonNull CameraDevice camera) {
Log.i(TAG, "Camera opened");
cameraDevice = camera;
createPreview();
}
@Override
public void onDisconnected(@NonNull CameraDevice camera) { }
@Override
public void onError(@NonNull CameraDevice camera, int error) {
Log.e(TAG, "Error opening the camera");
}
}, null);
}
catch(CameraAccessException cae) {
// Let us handle the error
}
catch(SecurityException se) {
}
}When the camera is opened and ready to use, the Android app can starts streaming images:
private void createPreview() {
SurfaceTexture surfaceTexture = preview.getSurfaceTexture();
surfaceTexture.setDefaultBufferSize(imageDimension.getWidth(), imageDimension.getHeight());
Surface surface = new Surface(surfaceTexture);
try {
previewRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
previewRequestBuilder.addTarget(surface);
cameraDevice.createCaptureSession(Arrays.asList(surface),
new CameraCaptureSession.StateCallback() {
public void onConfigured(@NonNull CameraCaptureSession cameraCaptureSession) {
cameraSession = cameraCaptureSession;
previewRequestBuilder.set(
CaptureRequest.CONTROL_AF_MODE,
CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
// Flash is automatically enabled when necessary.
previewRequestBuilder.set(
CaptureRequest.CONTROL_AE_MODE,
CaptureRequest.CONTROL_AE_MODE_ON_AUTO_FLASH);
// Finally, we start displaying the camera preview.
CaptureRequest previewRequest = previewRequestBuilder.build();
try {
cameraSession.setRepeatingRequest(
previewRequest, new CameraCaptureSession.CaptureCallback() {
@Override
public void onCaptureProgressed(
final CameraCaptureSession session,
final CaptureRequest request,
final CaptureResult partialResult) {
}
@Override
public void onCaptureCompleted(
final CameraCaptureSession session,
final CaptureRequest request,
final TotalCaptureResult result) {
}
}, backgroundHandler);
}
catch(CameraAccessException cae) {}
}
@Override
public void onConfigureFailed(@NonNull CameraCaptureSession cameraCaptureSession) {
Log.e(TAG, "Configuration failed");
}
}, null);
}
catch (CameraAccessException cae) {}
}Classifying the image captured by the camera
In this last step, it is necessary to capture the image and classify it using Machine Learning. To this purpose, the Android app uses the button defined in the previous layout:
labelBtn = (Button) findViewById(R.id.btnLabel);
labelBtn.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View v) {
takePicture();
}
});In the takePicture method, the Android app takes the picture and label it (as shown above). If you want to have the full code you can go to Github and download it.
Summary
At the end of this article, you hopefully gained the knowledge about how to build a Machine Learning Android app using Fritz.ai. You have discovered how to use Machine Learning image classification to label an image captured by the camera.
