How and why we created a chatbot for hiring ‘blue-collar’ workers
What is Avito Jobs and its audience
Avito Jobs is a Russian app and website to search and find a job or a worker: employers post vacancies and job seekers publish their CVs. Candidates can respond to a vacancy by phone or via in-app messenger. Employers can browse CVs while the access to candidate’s contacts is paid.
The audience of the app are ‘blue-collar’ workers: drivers, administrators, cashiers, movers, couriers, waiters, cooks etc.
Problem statement. Employers need to reduce time of primary validation of applicants
While working on customer journey maps of Avito Jobs users we conducted a number of interviews with both job seekers and recruiters.
One of the main problem was a huge amount of time recruiters spent on primary validation of candidates. Only very few applicants had CVs with essential personal information. The others had to be asked a number of questions via phone interview. Employers were eager to buy more CVs if possible.
From the business point of view that meant that our CV revenue stream had a huge room for improvement. The more CVs we have, the more we can earn selling them.
Unlike highly qualified specialists, ‘blue-collar’ workers usually don’t create CVs.
Quantitative data tells the same. Only 15% of active applicants on Avito Jobs have CVs.
At the same time employers (major retail and food chains, plants and factories, cell phone operators and real estate developers) are not eager to accept everyone. Some primary validation is needed.
In order to find out what exactly employers look for and what is the process of the initial assessment of candidates, I tried to get a real ‘blue collar’ job myself.
I decided to become a cashier. I applied to cashier position in 40 large and small organisations. After application I received calls from recruiters. All of them asked me the same:
— citizenship
— age
— gender (was actually obvious while talking)
— preferred job location, city and metro station
Why ask about preferred location? Because candidates usually want to reduce their way to work. They tend to choose something close to their home. Our quantitative research confirms this.
After a brief phone talk recruiters invited me to onsite interviews.
Solution hypotheses
How could we help recruiters spend less time on primary screening and find more suitable candidates? How can we encourage job seekers to share primary information about themselves?
We could make a CV creation form easier and shorter. We could split it into steps. We could reduce unnecessary feeds such as work experience, language skills and education.
We estimated the potential influence of this change. With a 20% increase in conversion to successful CV creation, the resume base would grow by a quarter.
Another way to provide recruiters with essential information on candidates was to change the way to collect that information. It could be done via chatbot, for instance. Then collected information could be passed to recruiters in form of CVs.
In order to estimate success of this possible solution, we assumed that half of those who received an invitation to talk to a chatbot would accept it. Three quarters of them would get to an end of the conversation. These assumptions were based on an experiment which Avito Jobs team made a month earlier using a bot in third party messengers.
In this case our resume base would grow by a little more than a quarter.
How we made a decision
As you can see the estimated resume base growth is almost the same for both solutions. This is why it was not only quantitative evidence which we based final decision on. We considered the connection of the possible solution to other strategically important initiatives to come. The chatbot could help us on our next steps of the hiring funnel:
— professional orientation help
— relevant vacancies recommendations
— onsite interview invitation
— getting feedback on onsite interview and getting a job
Why would job seekers want to talk to a bot
Getting back to job seekers behaviour, it’s important to note, that creating a CV has no clear value to them. The true is, when creating a CV, applicants could expect that employers would find and reach them out by themselves. This is what we decided to translate as the main product value.
Our further plans were to recommend vacancies based on the CV created. This could become an even stronger motivation to talk to our bot.
The flow
Based on my phone research I created a list of questions that out chatbot should ask:
- Preferred position
- Citizenship
- Age
- Gender
- Preferred job location
- Salary expectations (was later removed from MVP scope due to development time reduction)
- Phone number confirmation (this is important to make in order to reduce the number of occasions when a recruiter pays for the contact, but can’t reach a candidate out)
- Permission to publish a CV
The Interface
By the moment we started our research Avito already had its in-app messenger.
From the technical point of view the Messenger team provided a certain functionality scope, which could be used to base the development of a chatbot on. Unfortunately there was no functionality that could transform the user’s button choice into a message, as it usually works in chatbots. The chosen answer could be shown as a system message only.
This is how it looked like.
Such conversation required a careful work with wording. Our editor managed to pick the right words for each answer interpretation. Being a designer I could not stand the fact that the common messenger behaviour pattern was broken due to technical restrictions. As a team we faced the choice. We could launch chatbot with this weird type of answers and prove the usefulness of the whole idea. Or we could postpone the delivery in order to modify the transformation of user’s answers to proper messages.
We had absolutely no evidence that this transformation was crucial for users. The influence of this decision on the product was tricky to estimate and measure. In order to prove the opposite (the absence of influence) we conducted a UX-research. We showed the prototype to five kitchen workers form our office canteen and asked them to go though the scenario and create a resume. The goal of the research was to prove that people successfully finish the scenario despite controversial interface solution. To my surprise none of the respondents showed any discomfort when communicating to our bot. Thus we decided not to wait for interface modification and launch the MVP to the bot just as it is.
The research showed some more curious details. For example after picking location people expected to receive jobs recommendations immediately.
We also realised the we underestimated the importance of salary. We decided to add one more step to the scenario and ask about salary expectations. We also decided to show the salary together with each job recommendation.
Shortly after our UX-research the Messenger team unexpectedly changed its priorities and decided to make an answer transformation to a message. Neither we nor them still had no evidence this change was essential, but anyway this was built as the market normal solution. I’m still grateful to my colleagues for this.
But what were we to do with our research results? They turned out to be unnecessary. Let them become our experience for future.
First mini launch
In December 2019 we launched the first version of our bot on 1000 users who searched for jobs for drivers.
In order to speed up the launch we decided to postpone building recommendations. We received the results within a few hours.
1222 users received a push notification with an invitation to talk to a bot. A quarter of them started a dialog. 9,5% from all who received the push, went though all steps and created a CV. This was far below our expectations.
In order to find out why people refused to talk to our bot we called some of them immediately. There turned out to be three main reasons for that:
— Coincidentally people received job offers after receiving a push but before starting a dialog with our bot, so there was no sense to continue
— Some of them experienced difficulties to communicate due to poor Russian language skills. This is an important signal for us to think of localisation
— Misclicking
This is what we decided to do to give people an opportunity to get back to the conversation later. Our bot starts a conversation with a ‘Hello’ and a preposition to start a dialog. The first version included ‘Yes’ and ’No’ answer variants. We added one more variant — ‘Later’.
Problems occurred on further steps as well. This is what people answered to the question ‘What position would you like to work in?’
The thing is that these answers become the titles of CVs. First of all we reduced the allowed symbols number for this line. Then in order to solve the untranslatable problem with Russian grammar cases we changed the wording of the question. We added an example to it. And we also added a hint for those who were ready to work in any position.
There is no need to say the this wording caused an enormous growth of CVs with the title ‘Any position’. People found out this opportunity and started to use it. The liquidity of such CVs turned out to be the same as with regular titles, so there was nothing to worry about.
Second mini launch
During our second launch we addressed 5K users who responded to one of our popular positions: driver, administrator, cleaner, shop assistant, cook, courier and vacancies that didn’t require any experience and were suitable for students.
The results differed only a little. It turned out that opportunity to continue the dialog later didn’t help. People still didn’t want it.
The conversion to the start of the dialog grew only 3,5%, and the conversion to CV creation grew 5,5%. This growth was due to students, who are always eager to try something new. The couriers also contributed to the growth, because working with apps is part of their everyday routine. For us this is a signal to think of more intense approaches when addressing these audience segments.
The phone survey showed that in addition to previous problems people could not clearly understand the value of the conversation.
We decided to concentrate on the top of our funnel in order to widen it.
This is what we decided to do.
In order to reduce people’s time we decided to skip the request to take a couple of minutes and talk to our bot. Now we started a dialog directly with a question about preferred position.
To those who left the dialog on any step we sent push notifications to continue the talk. This notification turned out to be most effective for those who received it on the very first step. As long as a user started the dialog he was very likely to finish it.
And above all we changed the wording in the first push notification. We added more value to the message and excluded the poorly understandable word ‘resume’.
Third mini launch
These changes were successful. The conversion to start the dialog grew 7%. The conversion to CV creation grew respectably. These results were optimistic enough to start an AB test on statistically significant amount of users.
We expect the AB-test to show the growth of our CV base and CV revenue stream. These are the results we a going to base our next decisions on.
This is the start of a long adventure. We are determined to provide a candidates way towards his preferred job with via chatbot as an interface solution. I’m sure there are many more surprises and discoveries to come.
