Сегментация человека
Определение человека на фото

A neural network that segments mobile video frames and «searches» for a person on them

Application

Main information

Client
The developer of a popular mobile application for artistic processing of photos and videos.
Task
Create a tool that will segment the frames of the video stream and select the silhouette of a person.
The service must run on smartphones.
Application
The user of the application will be able to change the background, as in Zoom, or «put on» a mask.
Ranked in the top 10 on the App Store in the «Photo & Video» category
Decision
Three neural network models for a wide range of mobile devices. They work at different speeds and accuracy.

The problems we faced

Problem
There was not a single mobile developer on our team.
Decision
We have selected specialists who are well versed in both mobile development and machine learning.
Problem
The service had to work on different smartphones, including "slow" ones.
Decision
We have developed three models. The first is for «slow» smartphones, the second is for powerful flagships and the third, optimal, is for the main group of devices.

Development stages

We studied materials and tools for development
Picked up a library of videos and images
Trained the neural network to «see» a person in the frame and outline his contour
On the picture
On the video
Человек на лошади
Сегментация человека на лошади
Скейтбордист
Сегментация скейтбордиста
Сегментация человека в лодке
Человек в лодке
Группа человек
Сегментация группы людей
Сегментация человека на видео
Adapted the service to launch on mobile devices
Results
Original image
Perfect mask
The neural network has drawn a contour according to the probability that the pixels are related to the human figure.
Final outline. The neural network outlined it, predicting the position of the person in the frame.
Image
GT Mask
Predicted Probs
Predicted Mask
Изображение людей
GT Mask людей
Predicted Probs людей
Predicted Mask людей
We tested the neural network and passed it to the client
Тестирование нейросети
An example of testing a model on a Huawei P20 LTE smartphone

Tools

Vision Hub

Hosting platform for computer vision models. Models run on our servers and are embedded in third-party services via the REST API.
Платформа Vision Hub

Project team

Konstantin Kubrak
Project manager
Vladislav Vinogradov
Technical Director
Alexander Grashchenkov
iOS developer
Viacheslav Schultz
Сomputer vision engineer
Dmitry Gordin
Android developer

By the way

This was our first experience in adapting neural networks to run on mobile devices.
We handed over the development to the client and taught him to reproduce the effect on his platform.

Results

This project turned out to be more difficult than we thought. Thanks to him, we began to conduct audits and consulting in the field of machine learning.
The experience gained was very useful in another project - when we were developing an application for taking meter readings.

Plans

Make the model cross - platform - so that it runs on any device, as well as in the cloud.
Combine the segmentation model with the frame depth reproduction model. This will improve the segmentation accuracy.
Contact us