This tutorial describes how to create art with ESP32-CAM using style transfer and Magenta.js. One field where AI and machine learning can be applied is the art field. In this tutorial, we will explore how to modify the picture taken by ESP32-CAM using machine learning. This example demonstrates how we can mix different technologies to create something new and different. Moreover, this tutorial shows the endless possibilities we can have when using ESP32-CAM.
What is Neural Style Transfer?
Neural Style Transfer is a set of algorithms that manipulate images. It uses two images:
- a content image
- a style image
The aim of this process is to mix these two images together applying the style of one image to the content of the other one. The final effect is one image that holds the content of the original image painted with the style of another image. We will use the picture taken by ESP32-CAM as the content image while we will another image as a style image. We will mix them together. Neural Style transfer is used to create artificial artistic images. In this tutorial, we will experiment how to apply Neural style transfer with ESP32-CAM using Magenta.js.
What is Magenta.js?
Magenta.js is an open source javascript library built on TensorFlow. We have already covered how to use Tensorflow with ESP32 in the previous posts. This tutorial extends the concepts covered in the post “how to classify images with Machine Learning with ESP32-CAM” using the same process described previously. Magenta is an effort to explore how Machine Learning can be applied to art. As we did in the previous post, we can’t run this machine learning model directly on the ESP32 because it would require too much power. Therefore, we use the client power. In more detail we will use the browser because Magenta.js runs inside it. If you like to explore the power of ESP32 in Machine Learning, you can explore how to use ESP32 in Machine Learning with a KNN classifier.
More useful resources:
How to detect object using ESP32-CAM and Tensorflow.js
How to apply Style transfer to image using ESP32-CAM
To achieve our goal, we will create a simple HTML Web interface using the ESP32. This interface has three different tasks:
- Stream the video and capture the image
- Let the user to select the style image
- Apply Style Transfer using Magenta.js to the image taken by the ESP32
The UI interface is very simple. You can find more details about how to create it in the previous post. Moreover we will use the same ESP32-CAM code developed previously:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>ESP32-CAM Style Transfer</title>
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<style>
body {
font-family: 'PT Sans', sans-serif;
background: rgb(63,94,251);
background: radial-gradient(circle, rgba(63,94,251,1) 0%, rgba(252,70,107,1) 100%);
color: #636060;
line-height: 1.6;
}
a {
text-decoration: none;
color: #ccc;
}
h2 {
margin-left: auto;
margin-right: auto;
color:#ffffff;
margin-bottom: 10px;
}
h3 {
color:#ffffff;
}
.custom-file-upload {
border: 1px solid #ccc;
display: inline-block;
padding: 6px 12px;
cursor: pointer;
margin-top:5px;
background: #a944a6;
color: #FFFFFF;
}
input[type="file"] {
display: none;
}
button {
margin-top:5px;
}
</style>
<script src="https://cdn.jsdelivr.net/npm/@magenta/image@^0.2.1"></script>
<script language="javascript">
function loadImage(event) {
var selectedFile = event.target.files[0];
var reader = new FileReader();
var styleImg = document.getElementById('img_style');
styleImg.title = selectedFile.name;
reader.onload = function(event) {
styleImg.src = event.target.result;
};
reader.readAsDataURL(selectedFile);
}
function applyStyle() {
console.log("Applying style...");
const model = new mi.ArbitraryStyleTransferNetwork();
const contentImg = document.getElementById('img_source');
const styleImg = document.getElementById('img_style');
const stylizedCanvas = document.getElementById('stylized');
contentImg.onload = function() {
console.log('Capturing content image..');
model.initialize().then( function() {
model.stylize(contentImg, styleImg).then((imageData) => {
stylizedCanvas.getContext('2d').putImageData(imageData, 0, 0);
contentImg.onload = null;
contentImg.src = 'https://192.168.1.121:81';
});
});
}
contentImg.src = 'https://192.168.1.121/capture?t=' + Math.random();
}
</script>
</head>
<body>
<div class="container">
<h2>Magenta Style Transfer with ESP32-CAM</h2>
<div class="row">
<div class="col-sm">
<img id="img_style" width="320" height="200" />
<label for="stylefile" class="custom-file-upload">Select your style file
<input type="file" id="stylefile" class="form-control-file" onchange="loadImage(event)" />
</label>
</div>
<div class="col-sm">
<img id="img_source" width="320" height="200" src='https://192.168.1.121:81' crossorigin style="border:1px solid red"/>
<button type="button" class="btn btn-primary" onclick="applyStyle()">Apply style to your image</button>
</div>
</div>
<div class="row">
<div class="col-sm align-middle">
<span/>
</div>
<div class="col-sm">
<h3>
Your styled image
<h3>
<canvas id="stylized" width="320" height="200"></canvas>
</div>
</div>
</div>
</body>
</html>Code language: HTML, XML (xml)UI preview
This is the UI preview:
Installing the code into the ESP32-CAM
Clone the repository on Github and install it in your ESP32-CAM using the IDE you prefer.
Notice that the HTML page is in the hex format. This is a simple trick to avoid using a filesystem in the ESP32-CAM to provide the HTML page.
Applying Style transfer using Magenta.js and ESP32-CAM
Once you have installed the code into your ESP32-CAM device, we can test it connecting to your device using a browser.
http://192.168.1.121/artCode language: JavaScript (javascript)Below some examples:
Below another example:
As you can notice from the image above, the content image, that is the ESP32-CAM image, is transformed using the style of the another image.
Wrapping up
At the end of this post, we have learned how to use ESP32-CAM with Magenta.js. This is a javascript library to create art using Machine Learning. We have discovered how to apply Style Transfer to the image captured by the ESP32-CAM. We can modify these images applying the style from another image we select.
