How To Structure Data Science Teams For Maximizing Success
Whether we talk about startups or about new technology, the hype gets built because of a few big success stories. Something similar is happening with data science. We all know that data is at the heart of any company’s digitalization journey. We all know how powerful it is at providing insights which improve your operations, make customers happy, decide new products or find newer business avenues. Many people are jumping right into it. Although their intent behind starting data science projects is commendable, market studies show that 3 out 4 data science projects fail to deliver the desired outcome. I have been watching this space for several years now and have worked on several successful and a few not so successful projects. I know why this could be happening. Let’s analyze and understand how you can be a part of the few successful ones.
There are several factors that play a role. Unrealistic expectations, insufficient budgets and privacy issues are definitely some of the key aspects. Some of these issues can be avoided if one runs pilot projects to test the waters. You should invest more and go full force only when you see the potential.
The bigger issues revolve around the right team. As you will find soon, you truly need a multidisciplinary team. The nature of the beast calls for a team with different skillsets. As it starts from business and ends with RoI for business, you will need a business expert who brings in domain expertise. Secondly, there is a huge difference between ‘data’ and ‘right data’. ‘Data’ is just garbage. You need data engineers who collect the ‘right data’. You may need IoT experts in some cases to make this data available. They need to have the right rigor. You need data engineers and software engineers to put the IT elements and the code in place. Large volumes of data must be stored and processed really fast while maintaining the it’s security & integrity. Then in some cases, you may need digital signal processing experts to extract right features from the signals. The most important people in the project are the data scientists. They can decide which features of the data to use, understand patterns, take decisions on picking the right algorithms, choose machine learning models, train the models and guide the teams. They need to not only understand all the aspects of the data science process but also spend more time in higher order work.
Then comes the issue of MLOps. The models developed need to be deployed in real world scenarios. This is again a specialized field.
As you can see, this team needs to be multidisciplinary. Just because you are a data engineer, doesn’t mean it is fine to not understand ML or the business scenario. Yes, you need not be an expert in it, but you do need to have a broader understanding. The same logic is applicable to every stakeholder. It is hard to build team with deep expertise in the right areas and good understanding in others. There is a common perception that data engineers can be upskilled to become data scientists. While there is a merit in this argument, there is a difference between a mind-set of a scientist and an engineer. Unless the mind-set of performing experiments is adopted, it is an uphill task.
In this field, there is much lesser supply than demand. Subsequently, such a niche team comes at a cost. It took us more than two and a half years to recruit the right candidates and get them trained in these skillsets. What is the end result? Our success ratio hovers above 90%. We have the right team with redundancy in skills as well.
Then, we do something that is obvious. We start small and we make the smaller project successful. With the success, comes success. I mean money. Seeing is believing; Upper management tends to put in more money when they see potential impact. This also gives the team confidence that they are on the right track. Most importantly, the goals become more realistic. There is a reason why data science is at the top of Gartner hype cycle.
But, to get this smaller project to be successful, you need the right team. Data science is the right thing to do, but building teams should not be the first thing on your mind. It is natural to think that you could build your team first and use a consultant as needed. But that seldom works for companies because data science projects are multi-dimensional problems. Outsource the work to the right partner. You should stay involved as a business or domain expert. This not only increases your chances of success but reduces your fixed costs. And during the first run, you learn what kind of a team you need. Slowly, you can build your own team as your process evolves.
Simply collecting data and dash-boarding it is passé. The real power comes when you draw insights using data science. Let the machines do the hard work while you spend your time solving problems of a higher order. Taking the right decisions for growth and providing leadership in the right areas is where you should be focussed on. Leave the nitty-gritties of implementing data science to the experts.