A neural network that «searches» for people using mobile video segments
Application
Main information
Client
The developer of a popular mobile application for artistic processing of photos and videos.
Task
Create a tool for segmenting video frames and steam a person's silhouette.
The service must use a smartphone
The service must use a smartphone
Application
The application user will be able to change their background similar to the Zoom application or "put on a mask"
Ranked in the top 10 on the App Store in the «Photo & Video» category
Decision
Three neural network models for a wide range of mobile devices. They work at different speeds and accuracy.
The problems we faced
Problem
There was not a single mobile developer on our team.
Decision
We have selected specialists who are well versed in both mobile development and machine learning.
Problem
The service had to work on different smartphones, including "slow" ones.
Decision
We have developed three models. The first is for «slow» smartphones, the second is for powerful flagships and the third, optimal, is for the main group of devices.
Development stages
We studied materials and tools for development
Picked up a library of videos and images
Trained the neural network to «see» a person in the frame and outline his contour
On the picture
On the video
Adapted the service to launch on mobile devices
Results
Original image
Perfect mask
The neural network has drawn a contour according to the probability that the pixels are related to the human figure.
Final outline. The neural network outlined it, predicting the position of the person in the frame.
Image
GT Mask
Predicted Probs
Predicted Mask
We tested the neural network and passed it to the client
An example of testing a model on a Huawei P20 LTE smartphone
Tools
Vision Hub
Hosting platform for computer vision models. Models run on our servers and are embedded in third-party services via REST API.
Project team
Konstantin Kubrak
Project manager
Vladislav Vinogradov
Technical Director
Alexander Grashchenkov
iOS developer
Viacheslav Schultz
Сomputer vision engineer
Dmitry Gordin
Android developer
By the way
This was our first neural network adaptation experience to run on mobile devices.
We provided our developments to the client and taught them how to reproduce the effects on their platform
Results
This project turned out to be more difficult than we thought. Thanks to him, we began to conduct audits and consulting in the field of machine learning.
The experience we gained from this project proved useful in a later project - a application for meter readings
Plans
Make the model cross - platform - so that it runs on any device, and runs thru the cloud.
Combine the segmentation model with the frame depth reproduction model. This will improve the segmentation accuracy.
Contact us
PRODUCTS
SECTIONS
STUDIOS
Russia, Innopolis, st. Universitetskaya, 7
2023 © EORA. All right reserved