Neural network that adds animation and vibrant colors to videos

Analysis and processing of video streams

ESSA logo

About

01/

Client

Advertising agency APM is a partner of AB InBev Efes, a producer of beer and non - alcoholic beverages in Russia and Ukraine.

Background

ESSA ran an ad in which a person and their background were painted in different colors.

A task

Create a service where VKontakte and Telegram users will be able to create similar videos.
APM logo

What have we come up with

02/

A neural network that creates different video effects:

«Neon boom»
with the appearance of contrasting contours of the body, eyes and mouth, clothes
«Superwave»
with a wave filling the contour of a person
«Bright party»
with solid color filling of body parts, eyes, lips, clothes
Волна, заполняющая контур человека
Сплошная цветовая заливка частей тела, глаз, губ, одежды
Появление контрастных контуров тела, глаз и рта, одежды

What neural network can do

03/
Distinguish a person from furniture and background
Segment objects and paint them in different colors
Designate eyes, hair, drawing on clothes with different colors
01
02
03

How it works

04/
User reads the QR code on the ESSA beer bottle with his phone
The ESSA "VKontakte" service page opens on the phone
Чат-бот Essa
/start
*uploads video*
*selects video effect*
Hey! This is Essa's chatbot and I can animate any dance video of yours! Send me your incendiary video from 14 to 40 seconds long!
Video received! Choose a processing style for your video:
Neon boom
Superwave
Bright party
It will take me some time to process. I'll send everything soon.
Everything worked out! Your video is available at https: // ...
And here's how the neural network «saw» our team:
Нейросеть распознает несколько человек

Tools

06/
To create a bot, we used our own services:
Simple and convenient platform for developing chatbots, voice assistants and contact center automation systems.

zDialog

AI service for chatbots analytics. Allows you to conduct a deeper analysis of human-robot dialogues and get a more accurate assessment of the effectiveness of the bot.

OneDash

We create a hosting platform for the execution of computer vision models on any device. Models are executed on our servers and are available via REST API

Vision Hub

zDialog фреймворк
OneDash сервис

Development process

06/
Step 01
Trained the neural network to «see» and segment body parts and clothes on video
Step 02
Step 03
Step 04
We tested the service, made debugging
Step 05
Step 06
Wrote a user guide on how to best shoot videos
Launched the service
Trained the neural network to «see» and segment furniture on video
Made three different effects for the video

Difficulties of the project

07/
01
Problem
The custom videos were of low quality, with poor lighting.
Decision
To improve the picture, we used four computer vision models. They predicted the position of an object in the frame, making the image clearer.
02
Problem
The neural network could color different frames in different ways; noises appeared in the video.
Decision
We have improved the work of the algorithm that was responsible for the clarity of the picture.
03
Problem
Time-consuming video processing: a 15 - second video could take 15 minutes. There was a risk that the user would not wait or the servers would not be able to handle the load.
Decision
We began to indicate the position of the user in the queue to download the video, if it was necessary to wait. Added spare capacity for stable operation.

Project results

08/
A ton of new knowledge is our largest and most complex project related to image segmentation on video
01
02
The client was satisfied and decided to launch a similar service for users in Ukraine and Belarus

Project team

09/
2 project managers
3 backend developers
Technical Director
2 data analytics
ESSA logo

What have we learned

10/
Use complex combinations of machine vision models
Process video recordings of different quality, including compressed algorithms
Contact us