Data Engineering Podcast

Share on social media:

Support Data Engineering Podcast

[00:00:13] Unknown:

Hello, and welcome to the Data Engineering Podcast, the show about modern data management.   When you're ready to build your next pipeline and want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.   With our managed Kubernetes platform, it's now even easier to deploy and scale your workflows or try out the latest Helm charts from tools like Pulsar, Packaderm, and Dagster.   With simple pricing, fast networking, object storage, and worldwide data centers, you've got everything you need to run a bulletproof data platform.   Go to data engineering podcast.com/linode   today. That's l I n o d e, and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show.  

Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on.   The data you're looking for is already in your data warehouse and BI tools.   Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems.   No more scripts, just SQL. Supercharge your business teams with customer data using Hitouch for reverse ETL today. Get started for free at dataengineeringpodcast.com/hitouch.   Your host is Tobias Macy. And today, I'm interviewing Tim Freestone about Aluba, an assessment platform for evaluating data and analytics candidates to improve hiring outcomes for data roles. So, Tim, can you start by introducing yourself? Well, firstly, Thomas, thanks for having me. Great to be here. Yes. I'm Tim. I'm down in, I would say, normally sunny Sydney in Australia, but it's been pretty wet the last week. So  

[00:01:52] Unknown:

that's not a great advertisement for our travel industry. Yeah. So founder of Aluber. I've been doing this for the last 2 and a half years now. Prior to that, I was managing an analytics team at a company called HotelsCombined.   We're now part of KAYAK and Booking dotcom. That's really where the kind of origin story, I guess, of Aluber starts. And do you remember how you first got involved in the area of data management?   Yeah. So, again, back at atiles combined, now part of KIAC. Part of my role there was really around,   you know, managing our fairly small analytics team, which had some data analysts, SQL developers, data engineers, data scientists,  

[00:02:28] Unknown:

and that's really where, yeah, data management began for me. In terms of the Aluba project, you mentioned you've been running that for a couple of years now. I'm wondering if you can just give a bit of an overview about what it is that you're building there and some of the story behind how it came to be and why you decided that this was the area where you wanted to spend your time and energy. Yeah. So in my last role, I noticed a couple of big trends. Firstly, in terms of hiring,  

[00:02:51] Unknown:

I found trying to hire anyone in analytics   a real pain in the neck   in terms of how long the process would take,   how manual it was, how undata driven it was, and you know how much it was based on just gut feel and intuition.   And from my own perspective as a candidate going back the last 10 years, I can honestly say some of the worst moments of my life were trying to get a job. You know, the amount of kind of ghosting involved, lack of feedback,   you know, suddenly not getting a job you thought you were gonna get. Like, it just seemed like a really   unfair process,   heavily biased in the favor of employers.  

That was 1 1 big trend. Just hiring in general, I think, is is not done that   well. The other big trend I noticed was   data literacy   and the fact that, let's say, back at my last company I worked at, we had, you know, 6 or 7 out of a 150 of the company were data people.   But once you looked at what about another 30 or 40 people did day to day, like all the online marketers, the product managers,   even some of the engineers,   supplier managers, once you looked at their actual day to day tasks, a lot of it was basic analytics.   You know, it was running reports, sticking into trends,   understanding if, you know, metric a goes up and metric b goes down, that means this. So   this emergence of this basic data literacy skill set was clearly becoming very, very important, and so it was another   big piece to to Eluba.  

So fast forward to today, basically, we help businesses   both in terms of understanding the skills of their candidates,   most often for, like, analytics specialist roles like data scientists, AI engineers, those kinds of things,   and then also internally within business is help them understand their basic data literacy   often across a much broader array of kind of participants.  

[00:04:39] Unknown:

And as you mentioned, being a candidate and being on the hiring side is always a bit of a painful experience for everyone involved, and   that's kind of universal regardless of what role you're talking about. I'm wondering if there are any elements that are   specific to hiring for the data domain that are particularly difficult or that are unique to that overall set of requirements   that will allow you to focus on that for Aluba or anything that you're doing differently in Aluba versus some of the other hiring platforms that are out there, whether it's for software engineers or HR professionals   or health professionals, whatever the particular  

[00:05:17] Unknown:

job description might be. Yeah. Yeah. I think you're absolutely right. A lot of these problems are just endemic to hiring in general. I'd say for analytics and data, you know, there are   a unique set of skill sets or unique set of skills, I should say, which does crossover with engineering but ultimately are different. Part of the challenge, I think, is the combination of soft and hard skills.   With hard skills, I think, generally being   a lot more easily evaluated.   But soft skills,   most people would argue are at least as important,   but very hard to evaluate,   particularly in a kind of online environment.  

So you hear lots of businesses talk about the importance of cultural fit and those kinds of things.   I'm always kind of on the fence when it comes to this because, you know, you could say someone is or is not a good cultural fit for any reason, really.   And it's very hard to defend or measure.   So it's kinda like we have this gap at the moment in our understanding of how to assess these things,   which we need to conquer eventually. I don't think anyone's solved that nut, but I think in analytics in particular, this, yeah, this real combination of those soft and hard skills, hard skills being quite specific to analytics.  

[00:06:26] Unknown:

In terms of the Alooba platform, what are some of the overarching goals that you have for the business and for the overall impact that it might have on  

[00:06:36] Unknown:

the broader data ecosystem and community and the businesses that you're working with? Yeah. Great question. So our end goal, our end vision is a world where everyone can get the job they deserve.   I can say that   probably we're pretty confident we're a long way from that world. Okay?   We started within the analytics and data science space partly because I think that's where the biggest gap in the market was. Partly that's where my own skills and experience was. So, basically, we're focusing on that skill set and that area first within those 2 use cases I've mentioned, so understanding the team's capabilities   as well as understanding the strengths and weaknesses of your own candidates.  

And, typically, our product is used as what I call almost like a screening quiz. So it's really around   that very scalable, automated   objective measurement in a quick and simple way. So   there's definitely   elements that our product is gonna miss that you're gonna get from sort of more long form interviews and those kinds of more open ended assessments, if you like. And so our goal basically is to go   up and down the hiring process and productize all of it. So the moment we focused on screening quizzes within analytics and data science. We'll go up and down and really   decompose, I guess, you'd say, that whole hiring process to try to make it as data informed and data driven as possible.  

Interviews, I think, are something that most companies don't think a lot about but could be dramatically improved by the proper collection and measurement of data.   I think in the online hiring environment we have now, it gives us an unbelievable opportunity to be recording those interviews, gathering those data that we otherwise wouldn't have had in a physical environment.   So, generally, we're gonna go and build our products right throughout this process, and we kind of lump them together under what we call a little bit match. The other direction we may end up going is kind of across   the levels of seniority. So we've recently launched what we call a little bit junior, which is around understanding   data literacy, those graduate and intern   levels that businesses hire for. In terms of the overall  

[00:08:39] Unknown:

experience of people who are using Alooba, I'm wondering if you can talk through the primary challenges that the different cohorts face in   fulfilling their role in the hiring process, both the hiring managers and the candidates and the employees who are working with the hiring managers to evaluate the candidates and and the candidates who are trying to network with the companies and just that overall sort of end to end user experience?  

[00:09:02] Unknown:

Basically, the way our product works is we would work with the business, typically with the hiring manager as well as someone within the talent acquisition team. That'd be our our 2 main users.   And when we start working with a new business, we sit down with them, try to understand their hiring process. Like, what are the different stages that they have? What are each of those stages trying to actually gain, like, what what is the point of each of those stages.   And, normally, we try to help give them some feedback on how to make their process simpler, more data informed,   and also just generally more structured.  

What we find is when we sit down with companies, often they'll have steps of the process that don't really have a clear value add or they'll have multiple stages attempting to assess the same thing to try to help them understand, like, what are each of the stages and what are they actually trying to assess.   In terms of our products then, as I mentioned, it normally sits as like a screening quiz, so normally towards the start of the process, but businesses also use it further down. And we'll sit down with them and understand based on the skills required for that role,   what should the assessment look like. We try to make the assessment as closely aligned to that role as possible.  

Beyond that, then the business basically invites candidates to take the quiz on our platform   that may be automated through, like, an integration to their HR system, maybe manual depending on where they use it in the process.   Candidates then get an invitation from us. They go through and complete the assessment   on our platform.   Ideally, with the results being shown to them at the end, that's what we recommend for our customers, like a quick snapshot so that candidates can get some some meaningful feedback,   and then all the detailed results are kept within the platform for the business then go and make a new hiring decision.  

With normally being fairly simple, they'll just use the overall mark as a quick guide to understand who they should interview and kind of go from top to bottom. So, hopefully,   a more objective, measurable way   of getting that initial shortlist   relative to CV screening. And so that's really the main area where we see a big improvement is that CV screening step is so random and so biased depending on, you know, what side of the bed you woke up that day. You can either say yes or no to a candidate. We're trying to really overhaul that and get a much more objective measure in those initial screening stages.  

[00:11:16] Unknown:

As a candidate, it can often be intimidating   or off putting for somebody to give you some sort of a quiz or assessment, you know, whether it's you're early in your career and you're intimidated because you're not sure of   how well you're going to rank, or if you're later in your career and maybe you feel offended that they're trying to screen you with a quiz. And I'm wondering how you've approached the overall   experience from the candidate perspective to make it a positive interaction with the organization   and, you know, give them a good sense of sort of the value that   the   assessment is providing and maybe give them the ability to kind of reuse   assessments across multiple different organizations that they might be interviewing with.  



[00:11:57] Unknown:

Yeah. This is a very careful balancing act, and I think it's definitely something we haven't perfectly nailed yet.   So I can see,   occasionally, some kind of conflicts between what a employer might want and what a candidate might expect.   A big 1 would be,   like, anti cheating or cheating prevention.   So some   businesses we've engaged with have kind of expected us to have proctoring, like, some kind of live monitoring of the candidate   filming them in their bedroom or whatever, which we would never go down that route because that's just like a bridge too fast. So even if you would somehow reduce the cheating rate, it's it's definitely not worth it because it's just so uncannadate friendly.   In general, we've tried to make the experience as candidate friendly as possible. For example, we show candidates their results at the end. So even if you don't end up getting an interview with the company, at least you get this nice snapshot of your performance broken down by each of the skills   as well as this kind of list of what we call areas for review. So specific question topics that they might have answered incorrectly,   which they can then go off and do some research on.  

Next evolution of that kind of end of test screen is basically to have this curated list of articles for each of those topics. So we've kind of gone through the effort to go and find what we think are the best places to learn those things. So that's our general commitment.   I'd say, yeah,   definitely, there are some candidates who aren't gonna want to do an online test. I think definitely is an initial screener, and I think a lot of it depends on the type of company hiring, the role, and the kinda market conditions.   So I'd say in the current market conditions, generally quite a candidate driven market, and so you kinda gotta be careful to not deter candidates.  

Also, for really senior roles, definitely, they'll just be a set of candidates to say, yeah, I'm not I'm not doing this, which we get. So, like, businesses have to be pragmatic and, I guess, use Aluber in the best situations for them.  

[00:13:50] Unknown:

In terms of being able to   ensure that the assessment is useful in providing value, you need to be able to have some kind of universal rubric across the different candidates so that you can establish a useful baseline for   the organizations to understand how the various candidates might relate to each other in terms of their strengths and skill sets   and for the candidates to be able to understand sort of what is it that this organization is actually looking for as far as a skills capability. I'm wondering how you've approached that problem of being able to   set a useful and consistent baseline and have a way to be able to draw those comparisons   in a, you know, very practical and pragmatic way without bringing in any sort of opinions or biases into the equation?  



[00:14:36] Unknown:

Yep. So the way we approach this basically is that each assessment that a customer creates is customized for them,   choosing from our bank of 3 ish thousand questions,   then each candidate who takes that assessment basically faces the identical testing conditions in terms of the questions, the order of the questions, the amount of time that to complete it. Everything is kept constant, which then means that the results hopefully are quite easily comparable,   basically an apples for apples comparison.   Additionally, what we have, as you can imagine, is a lot of historical data around how candidates have gone taking those particular questions that are in any given test. So we have a pretty good sense of what a normal mark would be, let's say, an average mark for that exact assessment.   And we also know what our what we call the ELIBDA benchmark, which is like our predicted 90th percentile.  

So for a normal cohort of candidates   under normal circumstances, we predict, let's say, 65% will be that 90th percentile,   and that's the basic feedback that we give to customers.   Then in terms of the objectivity,   almost all our test types are automatically graded. So we've written the questions in a way such that there is a uniquely right answer   and that we can arrive at that automatically in in market   so that we kinda remove that subjectivity from the table. We have some more more kind of open ended questions where businesses sometimes use those. It's a bit of manual grading involved, but for the most part, 95% of the questions are gonna be automatically graded. So that's it's, you know, nice, consistent, and objective measure.  

[00:16:06] Unknown:

And as far as being able to   build out that set of questions and assessments, I'm wondering what you have used   to understand what are the useful pieces of information   to signal somebody's   capabilities   and skills for a given area   and be able to actually have a   useful sort of signal to noise ratio in terms of I'm able to answer these questions well, and so I am able to actually demonstrate that I understand this area versus   I'm just, you know, very good at taking tests, and so I'm able to, you know, make educated guesses and score highly versus not necessarily being an expert in the domain.  

[00:16:45] Unknown:

Yeah. Good question.   So, basically, the way we approached our content development was to try to take as   pragmatic an approach as possible. So we engage with data analysts, data scientists, data engineers   working across a variety of different companies and different industries and different countries   and basically said to them, okay. Well, what are the most important things that you use in your day day to day? So the most important skills, topics, concepts that are just absolutely essential to you doing your role as a data analyst, data engineer, data scientist.   So that was our kind of starting point. So I guess, like, the opposite of a textbook approach, we just went for what is actually practically used.  

We also had an understanding of what are the skills in demand   from customers based on all the job ads that are out there, you know, all those kind of skills that are listed and then all our face to face conversations with customers. So we got a sense of, like, the demand for skills   and then from both the kind of workers and then also the employers and built out that content.   So that's our general approach is to do it that way.   Additionally,   1 of the main pieces of feedback we try to get from our customers is, you know,   where are those false positives? So if candidates are scoring well on a test and then bombing out in an interview, that's like a really interesting data point for us to have. We collect that.  

I have to say, generally, that's where we're doing quite well, and that's what customers normally trial during a trial period is is is looking out for those kinds of things.   And I think assessment is kinda hard to game, I would say. Like, it's we like to call ourselves the jeopardy of the Luba quizzes. Like, we're just asking people knowledge   and fact based questions about their domains. So, you know, either you know it or you don't, basically.  

[00:18:27] Unknown:

As far as being able to   sort of give good coverage of different domains,   how do you work to kind of break down those questions or   give organizations who are putting together their specific skills assessment useful guidance in understanding which questions to select for   a particular   candidate role and making sure that you don't just kind of exhaust the candidate in, you know, saying, we're just gonna ask them all 3, 000 questions.  

[00:18:56] Unknown:

So   we basically onboard each customer 1 by 1, so we have a sales process. And during that process, we would normally help them build out the first few assessments for their first few roles.   Within the product itself, we also have these kind of guardrails, I guess, in terms of, like, how much time per question candidates have, how long the overall assessment is. We try to recommend that within a certain   zone that we know through data basically works pretty well.   In terms of skills to roles, the way we also do it is with our customers to kinda prefill our assessment builder using, like, a role based template   so that, for example,   it would be very hard for a customer to give a BI developer   some deep learning questions as an example, which would kind of be off point. Yeah. So we kind of help them to do that.  

Also, we're helped by the fact that the main users we recommend to go on and kind of maintain those assessments and build them out in the future   are the hiring managers themselves and then talent acquisition who might not have the kind of deep market level knowledge are more responsible for kind of the candidate administration and those types of things.  

[00:20:03] Unknown:

And so digging more into the actual Aluba platform, I'm wondering if you can talk through some of the ways that you've designed the system to be able   to manage the   assessment tracking, the data points that you're collecting, some of the analytics that you're using internally to understand how effectively the system is working, some of the data collection that you're doing with the organizations that you're working with to feed that back into your system, and just some of the kind of iteration cycles that you've gone through to be able to build out the platform,   verify its effectiveness, and then improve in the areas that you've identified weakness?  



[00:20:37] Unknown:

So I guess we get a few interesting datasets that we are using at the moment to make most of our decisions. So   1 is   just face to face conversations, face to face. Let's say face to face in the world of a pandemic. So video to video,   that is,   I think, the most valuable   dataset for us, like, actually speaking to customers   as part of the sales process, the onboarding process, catching up with them once they're starting to use the product. Like, there   is no substitute for that direct feedback,   and I think no level of clickstream analytics will ever give you   as many insights as just speaking to your customers. So that's fine.  

Second part is we use, like, a UX   online testing platform whereby we can expose our product again and again and run tests and trials on, basically, our target user target market   of people who've never actually seen our product before. So, like, getting in in front of this fresh set of eyes again and again, and that really helps us with a lot of kinda UX types of issues.   We get a lot of really useful feedback directly from candidates. So at the end of the test, we basically ask candidates these set of questions as well as an open ended set of questions as well, and so that's been, like, a real treasure trove of data for us to continually improve the candidate experience   and making it as kind of simple to understand as possible.  

And then we have our own product analytics around, you know, what people are clicking on and where they're landing and those kinds of things as well, which has been really, really helpful.   But, yeah, definitely, the direct from the horse's mouth customer based feedback has been, I guess, the most informative,   although the most unscalable.  

[00:22:16] Unknown:

And then in terms of the   sort of design and goals of the system, I'm wondering how those have changed and evolved from when you first began working on it to where you are today   and how the sort of user interactions and user interviews that you've done have helped to guide that evolution.  

[00:22:33] Unknown:

Definitely, the original use case that we had was more around the hiring use case and understanding,   you know, candidate skills that are very easy to understand than us. That is what we've done. But what we now call a little bit growth, which is understanding your team's capabilities has ended up really kinda taking off more than what I would have thought. And   so there's a lot of crossover between those 2 things, like understanding the skills of the candidate and understanding the skills of participant in your business.   But then there's a lot of insights and a lot of, I guess, ways of presenting the data which can be quite different depending on those 2 use cases. And so we've gone off and kind of built out separate dashboards and front ends to display   those insights in a different way for those different target audiences.  

Also, we found generally when   assessing people internal to a business, it's a little bit different in terms of wanting to keep the intimidation factor down as far as possible,   wanting to kinda get that buy in and making sure that everyone within the organization knows why they're taking this Eluva quiz and what the benefits are and, like, really a lot more of a sales job on terms of the why, which has actually, I think, helped us on the hiring side indirectly because we've kind of thought more about selling this from a candidate's perspective and making it as as kind of friendly as possible for them as well.  

[00:23:46] Unknown:

Given that you're building a business that is aimed at helping other businesses hire people in the data and analytics space, and you're using data analytics to run your business. I'm wondering what are some of the ways that you've actually used Aluba   to help build Aluba?  

[00:24:01] Unknown:

Yeah. Well, I mean, for our own hiring straight off the bat, so at least for the last year,   we've used Aluber even for all our software engineering hires, and that content that we built out for our own hiring has actually become really useful because what we've noticed is that a lot of data engineering roles are almost   bordering on a software engineer   with a kind of data specialization. So a lot of that content that we built out more for our own internal hiring has actually been really useful in the product.   The other   big areas which we're seeing internally is more around   kind of validating   growth in someone's knowledge.  

So I I like to say at the moment in life, we have not so much a knowledge gap but an action gap   in our decisions on what we do day to day. And so what we're quite big on here is, like, you know, for performance reviews and performance plans, setting setting about goals and being able to measure whether or not those goals have been attained. And so I think an alluvial quiz is a great way to do that internally by being able to say you're at x level and now you're at y, and that was, you know, your goal to get there over the last 6 months. And so that's, yeah, also   a way we're using the Looper internally.  



[00:25:10] Unknown:

Struggling with broken pipelines,   stale dashboards,   missing data?   If this resonates with you, you're not alone.   Data engineers struggling with unreliable data need look no further than Monte Carlo, the world's first end to end fully automated data observability   platform.   In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines.   Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes.   Start trusting your data with Monte Carlo today. Go to data engineering podcast.com/montecarlo   to learn more. The first 10 people to request a personalized product tour will receive an exclusive Monte Carlo swag box.  

So you mentioned early on that beyond just being able to evaluate candidates, you also have a system for being able to evaluate   understand   how you might be able to leverage the data assets that you have within the business, some of the types of trainings that you might want to do for people who aren't specifically   a data focused role. I'm wondering if you can just talk through some of the types of assessments that you're doing there to be able to establish that baseline and some of the ways that that information can be used within an organization.  

[00:26:35] Unknown:

So businesses engage us typically   when they are   thinking of   training and learning and development.   So they're at the stage of going, okay. Well, we know we have this budget. We know we're gonna wanna train people in something, but, like, train them in what?   So   that is   the ideal use case of Aluber is to instead of just saying, hey. Like, everyone's gonna get this training, but why not collect some data first and understand where the actual strengths and weaknesses are and then use the budget to train people where they're weak,   a. B, after the training's done,   reassess people to understand whether or not the training was actually effective.  

So it's really the 2 main trigger points is getting that initial baseline   and then measuring the efficacy of that training program.   And that has been most often done   in, yeah, broader data literacy   skills. So not necessarily   for analytics specialists,   but that kind of basic skill set that almost anyone would need in any role across a business.   That's generally where we've been engaged.  

[00:27:40] Unknown:

So for being able to   evaluate the data literacy of an organization, you know, 1 of the   terms that has become a bit buzzword worthy lately is the idea of data literacy   and businesses being data driven. And so sort of in order for organizations to be able to realize that goal of being data driven, you need to understand how to use the data that you have in order to actually   produce more value as an organization.   And I'm wondering if you can talk to some examples of companies that you've talked to where maybe they had   a low level of   sort of overall data literacy,   some of the steps that they've been able to take after evaluating their organization to be able to sort of raise that level of literacy across the business?  



[00:28:27] Unknown:

For us, it's mainly around that kind of measurement piece. And then the way   our customers deal with that is to put in place those kind of customized learning and development plans   and then,   remeasure   how well they've gone at kind of teaching their cohorts, whatever those skills gaps   were. Beyond that, that's not really something we get involved with in terms of, like, the actual implementation.   We leave that to the to the consultants.  

[00:28:51] Unknown:

So there are sort of 2 sides to the overall hiring process where you've got the candidates, you've got the hiring organizations,   and there's always a lot of kind of fear, uncertainty, and doubt involved in that process from both sides.   And I'm curious what are some of the   useful   hiring practices   and candidate sourcing techniques that you've seen to be able to improve the   transparency of the process so that both sides can get a more positive experience and a more beneficial outcome?  

[00:29:22] Unknown:

Transparency is a huge 1 because I think the status quo in hiring is complete opaqueness   with the employers basically holding all the cards and not really revealing anything.   So   what we envision, which is not necessarily what's done at the moment, is   as a candidate is applying for any role that a business would incredibly transparently   lay out what all the different stages are, a,   what the intent of those stages are in terms of what's the benefit to the company, like, why are they doing a stage? Like, you have this interview. What is this interview about? Is it about soft skills? Is it about hard skills? Are you trying to ascertain x, y, zed?   So laying out very transparently all those stages   and how long they'll take in terms of time coming in from the candidates and how long the overall process will take. I think that's where we need to get to. To be honest, I haven't actually seen any employers   lay it out in that level of transparency,   at least not at the application stage. Sometimes when you speak to someone for the first time, they'll kind of run you through those steps, but that kind of transparent,   here's everything that's gonna happen, is not generally done at all, which I think is where we need to take hiring, and that's really what we're gonna look to productize over the next couple of years. It's just that very transparent process   where we can have all these different stages, measure each stage,   understand how long each stage is gonna take, who's involved, why.  

Part of the issue, I think, why companies don't do this at the moment is they haven't   thought that far ahead often. Like, so many times,   the goalposts will be moved during the hiring process, and there will and won't be people available for different things, and they'll kind of add in stages.   And we know this ourselves because we kind of run a recruitment   service, so we'll be seeing our companies just chop and change the the drop of a hat. So I think we need to get to that point where hiring has been thought of so systematically   that each of the stages can be kind of   set out in stone,   measured properly, automated where possible, and we can get this beautiful transparent measurable process whether we get treated fairly and objectively.  



[00:31:25] Unknown:

And another problem that happens a lot, and it happens in many engineering roles, but definitely very much in data where you don't necessarily   know   what you're looking for in the role. You just say, I need somebody to do everything. And so you don't exactly know which skills to assess for,   especially if you're a smaller organization, and you're trying to just hire 1 person to do it all. And I'm wondering what you have seen as some useful   methods to help break down the actual requirements for an organization   and the skills that they need the candidates to have or being able to understand   when all of those skills might be able to coexist within 1 person versus needing to say, no. We actually need to hire 2 or 3 or 5 people to be able to fit all of these different use cases and needs across our organization?  



[00:32:15] Unknown:

For a small business, that is tricky. You don't have the existing   skills and knowledge internally. Like, if you were hiring your 1st data person, you weren't a data person yourself, that is tricky.   I feel like, you know, the concept of a data person as a unicorn has kinda died by now. Maybe a few years ago, it was more   the fact that you had to somehow know machine learning and build data pipelines and visualize stuff and do presentations and all that jazz. I feel like that's kinda gone now,   so it is gonna be tricky for any business if they don't have those skills internally.   I think you can kind of look to the market. Like, if you just look at 20 job ads for data analysts or data scientists or data engineers, you'll get a sense of   what at least those titles normally convey in the market on average to the typical business for the typical candidates, and you get a sense of where you're likely to find those skills.  

But, yeah, other than that, I would be, yeah, getting some, I guess, guidance from any colleague you have who are, experts in those areas to know what skills tend to go with   other skills and what skills generally don't, where you're likely to get your kind of best pen for your buck. That's where a good recruiter can come in, actually.  

[00:33:25] Unknown:

The other challenge in   understanding what skills you need is the fact that the data ecosystem is so rapidly evolving. And so what might have been a useful set of skills 5 or 10 years ago   is no longer directly relevant because of the shifting technology landscape and the availability of services where, you know, 10 years ago, you needed a Hadoop expert who knew everything about distributed systems and MapReduce. And today, you need somebody who understands how to connect Fivetran and Snowflake and DBT to be able to build out your semantic layer to power your data warehouse. And, you know, there's definitely some level of overlap in those capabilities,   but there's definitely a lot of kind of disjoint skills and understanding that's necessary where, you know, maybe now it's more important to be able to communicate effectively with the business to do requirements gathering and help them build out the semantic layer versus   10 years ago, it was a very engineering heavy capability of being able to   build and tune distributed systems. And, you know, you're working with the business, but not at quite so high of a level. And so I'm wondering if how you're managing   the kind of constant upskilling and evolution of skill sets that are necessary to work effectively in this kind of constantly shifting technology landscape?  



[00:34:33] Unknown:

Yeah. This is a tricky 1. I think there are a staggering number of tools now out there. I mean,   really an amazing number.   We've generally tried to approach this by   not going with, like, tool specific   questions and assessments, but rather trying to focus on, like, the core concepts and keeping it tool agnostic.   Because   the evolution is so quick, I don't think we could actually keep up. We would need such an enormous number of questions in our bank. It would it would kinda crush us.   And the other thing is we've generally found that   businesses   were happy to   hire candidates   without   experience in tool x as long as they had experience in a tool like x, if that makes sense. So as long as they could visualize data, they didn't care whether it was in Tableau or Power BI.  

As long as they'd written SQL, they didn't care whether it was MySQL or SQL. So tried to keep everything   kinda like tool agnostic   to then make it, yeah, more scalable and also be able to give every candidate the same opportunity   rather than forcing candidates to, let's say, here you go. I use this visualization in Power BI. Like, well, I've never used that, but I've used Tableau for 5 years.   Clearly, they could pick it up pretty quickly if if they wanted to, you know, within the confines of the time test is gonna be quite hard. So, yeah, we try to keep everything kinda tool agnostic. That's our approach to that. In terms of the bank of questions  

[00:36:00] Unknown:

and widely they're being used or how effective they are in being able to vet the candidates, I'm wondering what your process is for understanding   when a question is actually   not helpful or it provides a   negative signal in the hiring process and the kind of utility of being able to retire those questions and   understand why they're not useful and inform the future creation of questions as you continue to build and evolve the platform?  

[00:36:26] Unknown:

Yeah. So I think our, let's say, content analytics is reasonably nascent. I think there's a lot of areas of improvement for us there. A couple of data points we look at at the moment are things like so we have this question flagging feature. So Canada can flag a question within a test or report   it, and then at the end of the test, they'll give us a bit more details on why they reported it. So that's just, I think, the most useful direct,   set of feedback from candidates. That could be any number of things. It could be they thought the question was confusing, poorly worded, maybe it had a spelling error, maybe they didn't think it was relevant. It's basically up to them. So that's a very useful dataset which we use to either modify or, as you say, retire questions.   Another kind of more automated metric we collect is   the correct answer rate.  

So   questions that almost every candidate gets right almost by definition are pointless because they have no information.   Like, if the question was, you know, what color is the sky? Blue, green, red, yellow. Everyone clicks blue. It's just a waste of 30 seconds of a question. We could have included something else. So   questions with a really high correct answer rate, we get rid of.   Really low correct answer rates is quite an interesting problem because I think it could indicate 2 different things. 1 is   either a question that's quite confusing, like, if almost everyone gets it wrong, like, something unusual about that,   or it could be a really, really good question   that differentiates   between someone who has really expert knowledge and good knowledge.  

So for those kinds of questions, we put more into, like, a manual review process to go through and and check out and see, okay. Is this question   is this question wrong, or is it just actually a really good question we need to keep in? So there's an interesting conflict there. The other more kind of anecdotal feedback we get is just directly from customers and hiring managers themselves   when they either tell us about specific questions   or engage with our question bank feature and add their own content within the platform.   That's a really interesting insight for us in terms of, like, well, why are they adding all these questions? What are the gaps in our content that means they would need to add their own questions? So that's also really useful bit of feedback for us. In terms of the  

[00:38:34] Unknown:

reduction of bias and increasing equity in the hiring process,   you know, know, it's definitely useful to have an anonymized   kind of question and quiz interface so that you can have a consistent baseline. But then there is also still the implicit source of bias in that not everybody has equal access to Internet or time availability to be able to actually   complete an assessment of this nature. And I'm wondering how you think about that aspect of equity and inclusion   and some of the ways that organizations   might be able to   help in either providing them access   and time as part of the interview cycle to be able to conduct the assessment or   ways that you as a platform can maybe give people ways to chunk up the time that they spend on the questions to distribute that time requirement   in a way that suits their schedule better and just some of those aspects of the   kind of inclusiveness of the hiring process?  



[00:39:30] Unknown:

Yeah. Great question. I think this is something, again, we've   only partially solved and touched on and generally have left up to the behest of our clients that I think we can improve a lot on. In terms of our own platform, so, yeah, we have these kind of quizzes   that even if they're part of an overall assessment, can kinda be broken down and done separately at separate points in time. So, typically,   as long as a candidate can carve out half an hour, that should be sufficient to sit down there and focus on it. Customers can also allocate an extra time to candidates on, like, a case by case basis.   It does start to get tricky because we're also trying to keep this kind of level playing field and have all the conditions identical for everyone, but then some candidates gonna need   extra time to kind of get back onto the level playing field. So this is something that I don't see an obvious right answer for. He has to of the clients on how they do that. And then I guess the other aspect is just trying to think about,   like,   our platform and our approach relative to the status quo,   which is   CV screening   with someone looking at a CV for 5 seconds and basically saying yes or no in a CV, which contains, you know, so much personal information about a candidate that is   obviously the known cause of bias,   And so we try to compare ourselves against that incumbent process   rather than, you know, a perfect world that we're trying to create, but definitely aren't there yet. And in terms of the  

[00:40:56] Unknown:

applications of Aluba and the organizations that you've worked with, what are some of the most interesting or innovative or unexpected ways that you've seen it applied?  

[00:41:04] Unknown:

I have to say things have generally   panned out the way that we thought they would. I remember in the early days of researching Aluber, I spoke to a few hiring managers who said, oh, you know, when I took over x y zed function, I would have loved this so I could assess everyone and then get rid of all the people who had poor skills off the bat. I didn't have to wait a year to figure it out.   Thankfully, that's not been the main use case of Eluver so far. It's been a bit kind of friendlier than that. So I have to say, yeah, no huge surprises in the way it's been used either to assess candidates or to assess teams.  

[00:41:37] Unknown:

And in your own experience of building and growing the Aluba platform, particularly given that you started it around the same time as the pandemic, I'm wondering what are some of the most interesting   unexpected or challenging lessons that you've learned in the process?  

[00:41:49] Unknown:

You're right. We've basically scaled   throughout the pandemic. Like, we raised our seed round   literally the day that Sydney was going into lockdown last year, so it was just perfect timing.   I'd say the biggest challenge has been team building, actually, and scaling out a team   of people working in their bedrooms in, like, 8 different time zones in 8 different cities throughout the world and having that kind of togetherness   and and building that culture of the business remotely   has been incredibly hard. I don't think we've quite figured that out yet. So, generally, team building, I'd say, has been the biggest challenge.   There have been actually some opportunities in a sense that the pandemic has thrown up because   because everyone's doing sales calls and onboarding remotely. It's kinda leveled the playing field, and the fact that I'm sitting in Sydney and not in San Francisco doesn't really matter because I have the same opportunity as anyone else. So there's actually been some upside. So I think we've gotten some global expansion quicker than what we otherwise would have   because, yeah, having sales calls online has just been completely normalized.  

So, yeah, definitely some challenges, but also some some upsides as well.  

[00:42:54] Unknown:

For organizations   who are looking to hire on data professionals and they wanna be able to set an equitable baseline across their candidates or for candidates who are looking for jobs in the data space. What are the cases where Aluba is the wrong choice and they might be better suited either building out some in house set of assessments   or tailoring their interview process or for candidates just, you know, rely on the   time honored approach of just networking with the different companies?  

[00:43:21] Unknown:

Yeah. I'd say Olivia would be the wrong choice in a couple of scenarios. 1 would be really senior roles.   So if you're hiring maybe, let's say, ahead of   the only reason I say that is not because we wouldn't be able to derive insights, but because   there's just gonna be an expectation from candidates, they wouldn't have to complete the test. Like, there's a certain level   that you get to where you just don't expect that. You know, you wouldn't give a CEO of a bank test. Right?   So that's just the market conditions rather than the lack of insights, I think.   Likewise,   any roles   where there's a very tiny number of candidates for whatever reason, so it might be really senior roles or if you're in a market where there's just no candidates available,   then   I wouldn't necessarily use the Libra in the same way that some of our other customers do.  

Likewise, if there are roles   within data, not that I can think of any, but roles that are mainly soft skill based roles,   like a a sales role. I find it very hard to understand how we would assess for something like that, But as long as the role's within analytics and has that core kind of fundamental skills that I think could can be used reasonably well.  

[00:44:26] Unknown:

And as you continue to build and grow out the Aluba platform, what are some of the things you have planned for the near to medium term or any projects that you're particularly excited for?  

[00:44:37] Unknown:

Yeah. So I think 1 of the most consistent   bits of feedback we've gotten from the market, let's say, is that   businesses are   often more interested in a more holistic   solution to their problem.   So,   yes,   our kind of screening quiz platform solves the problem as part of the hiring process, and for bigger organizations, that's quite a big problem, especially when they hire tens or hundreds of roles a year. Like, we've solved a problem for them.   But then there's a lot of other steps of that hiring process in terms of sourcing candidates, other screening,   parts like the interviews,   and all those types of things. And where we found a lot easier traction is with what we call the match, so like a a matching service for analytics and data science roles.  

So we are basically gonna start productizing   that whole process in our platform.   I think that's really where the huge growth is over the next few years   is in, you know, setting up a way for us to set up interviews in our products, to record those interviews, to have a proper   measuring rubric for those interviews,   and then a much more holistic assessment of candidates. So not just the kind of screening quiz, but also these interview processes,   and then automating all the time wasters   in the hiring process. It's like interview scheduling.   God, that's just such a waste of time, like, back and forth between candidates and hiring managers and whatnot. So all these different basic stages of that hiring process, which we can either   automate or at least embed into our product,   measure the right things about that stage of the process,   and then roll out, you know, basically what we what we call the match. And that's that's what we're gonna focus on next year. And are there any other aspects of the overall process for  

[00:46:22] Unknown:

being a candidate to a data position or hiring for data professionals   and the ways that you are   using Aluba to help to   smooth that overall experience that we didn't discuss yet that you'd like to cover before we close out the show?  

[00:46:36] Unknown:

I think,   again, just to touch on a little bit match for a second. So this is a recruitment service we run for analytics and data science roles,   and so very much powered by our core assessment platform in the way that we can assess candidates at scale.   But then we've kind of matched it with   the kind of high touch interview process and requirements gathering with the business and those kinds of things. So it's like a recruitment service but data driven 1, and that's really where we're gonna take the product over the next year. I feel like that solved   quite a few other problems for customers in addition to the skills assessment.  

1 is mainly because   when we do that, we source candidates very, very widely and very aggressively, very wide top of the funnel,   and we funnel all those candidates through our assessment quiz, which is nearly scalable and quick.   And then we interview them with an expert for that role. So by the time you present a candidate to a client, they're kinda prequalified in a way that's atypical for traditional recruitment companies.   And that's, yeah, where we're getting a lot of traction there is that more holistic   recruitment picture, which we're gonna automate and productize over the next 6 to 12 months.  



[00:47:47] Unknown:

Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as a final question, I'd like to get your perspective on what you see as being the biggest gap on the tooling or technology that's available for data management today.  

[00:48:01] Unknown:

Data dictionaries are always something I hear about but never actually see implemented well.   So   the amount of upskilling and knowledge   that someone has to have coming into an organization and getting exposed to   the data, be it in a warehouse and dashboards and here and there and everywhere,   like, if there's gonna be some way holistically to solve that in the future where anyone coming on board just gets this quick snapshot, here's all these definitions, all these things, here's how they work, here's how you access data and x, y, and zed, like an actually   well   articulated   and   embedded data dictionary, I think that would be   an amazing   product to have and to see in the marketplace.  



[00:48:45] Unknown:

Alright. Well, thank you very much for taking the time today to join me and share the work that you're doing at Aluba. It's definitely a very interesting platform and   interesting approach to a very real and   necessary problem. So I appreciate all the time and energy you're putting into helping to make hiring for data roles easier and   a more positive experience for everyone. So thank you for that, and I hope you enjoy the rest of your day. Thanks so much. Thanks for having me.   Listening.   Don't forget to check out our other show, podcast.init@pythonpodcast.com   to learn about the Python language, its community, and the innovative ways it is being used.  

And visit the site at dataengineeringpodcast.com   to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned   show, please leave a review on Itunes and tell your friends and coworkers.