Chatbot to help women find gifts for men

Image analysis

AVON logo
Тимур Родригез с розой в руке
ABOUT
ABOUT
ABOUT
ABOUT
ABOUT
ABOUT

Main information

01/

Customer

Create a picker of gifts through «Vkontakte»

Task

Bot function

Girl uploads a photo of man to the chatbot
Project within the Avon advertising campaign by February 23
AVON

Audience

Women 25-45 years old

About the project

The bot helps with the choice of a gift depending on the type of man

Project character

02/
As part of the project, showman Timur Rodriguez tried on the role of a professional translator from male language to Russian.
Стикер с Тимуром Родригезом
Avon Россия, чат с Тимуром Родригезом
«Привет! Я Тимур Родригез. Я профессионально перевожу с «мужского» на русский. И помогу тебе выбрать лучший подарок для твоего мужчины»
After the girl uploaded a photo of the man, the bot made comic comments on behalf of Timur:
Шутка от бота Avon
Avon шутит про мужчину
Avon шутит про фигуру мужчины
Avon шутит про головной убор мужчины
Тимур Родригез смеется до слез
Тимур Родригез смеется до слез
Тимур Родригез смеется до слез
Тимур Родригез смеется до слез
The chatbot distributed original gift ideas based on the data received about a particular man
artificial intelligence with Timur Rodriguez
Чат-бот выдал идею подарка для мужчины

Project team

03/
Techlead
Vladislav Vinogradov
Chatbot developer
Roman Afanasiev
Project manager
Emil Maharramov
Data Scientist
Ramil Gizzatulin
Konstantin Kubrak
Computer Vision Engineer
AVOB logo

Development duration

04/
Integration
Markup
Validation
10.2%
13.6%
20.3%
Preprocessing
10.2%
Testing
8.1%
Deploy
5.1%
Training
32.5%
Total:
3.5
months of work
Seven neural network models are used for image analysis

Interesting fact

Work scheme

05/
Girl uploads a photo of a man to the chatbot
The neural network analyzes the photo according to the parameters:
01
Type of clothing
02
Shape type
Hair color
03
Hair type on the face and head
04
Мужчина в пальто с портмоне
The neural network creates a segmentation mask:
Маска сегментации
fair-haired
mustache
stubble
thin
business
Depending on the parameters obtained, the man in the photo is assigned one of four types:
Brutal
Homebody
Metrosexual
Fidget
Брутальный мужчина
Мужчина домосед
Мужчина метросексуал
Мужчина непоседа
Each type has its own recommended gift - Avon cosmetic product

Tools

06/
To create a bot, we used our own services:
A simple and convenient platform for developing chatbots, voice assistants and contact center automation systems.

zDialog

AI service for chatbots analytics. Allows you to conduct a deeper analysis of human-robot dialogues and get a more accurate assessment of the effectiveness of the bot.

OneDash

We create a hosting platform for the execution of computer vision models on any device. Models are executed on our servers and are available via REST API

Vision Hub

zDialog фреймворк
OneDash сервис

Some statistics

07/
photos
175 435
audio
78 434
During the campaign
2 343
messages

Technical implementation

08/

revealing gender of a person

01
Since the project was designed only for men, it was necessary to teach the neural network to determine gender.
To do this, we took a large dataset consisting of photographs of people with indication of gender.
To increase the accuracy, we used the dlib neural network, which detected the face on the photo.
Схема определения пола человека с коэффициентом

clothing analysis

02
To analyze clothes, we also used a segmentation model, that is, the selection of clothes, and then assigned it one of five classes:
In addition, it was necessary to teach the neural network to find nude photos, i.e. lack of clothes in the photo.
Схема определения вида одежды человека с коэффициентом
casual
sportswear
military uniform
business clothes
extreme outfit

figure analysis

03
In order to determine the type of person in the photo more accurately, it was necessary to train the neural network to recognize the figure.
We decided to take two main parameters - to learn to understand the complexion of a person and his approximate weight.
So we were able to separate four classes:
thin
full
normal
muscular
Схема определения фигуры человека с коэффициентом

analysis of vegetation zones on the face and hair color

04
First, it was necessary to determine the type of vegetation by zones or its absence, and then determine the color of the hair / beard.
To do this, we also needed a face detector, and in the next step we used a pre-trained Tiramisu model and a set of rules to make the detection more accurate.
After that, it was necessary to determine the hair color:
Классификация цвета волос, список цветов волос

background analysis

05
The last task was to classify the background - it was necessary to try to determine the shooting location or characteristic features such as the sea, a car, and so on.
Due to the low quality of real data, the result was not very accurate, so it was practically not taken into account in the final model.
Схема определения фона на фото
An even more detailed presentation of the project can be found here

Project difficulties

09/
«Oddly enough, there were almost no difficulties. The hardest part was to classify the background. We spent with it a lot of time, but as a result we just decided to remove it - it turned out that the other parameters were enough».
Konstantin Kubrak
Computer Vision Engineer

Project results

10/
It was interesting and exciting, because many things were new:
The first chatbot to use machine vision
A pipeline of seven neural networks within one project
First attempts to analyze sound
Contact us