Developing a Robust Data Strategy
As the cloud opens up new possibilities for incorporating machine learning and artificial intelligence into your offerings, it is rapidly becoming essential to develop a robust data strategy. Deep learning and new breakthroughs in AI are extremely powerful, but they require large data sets in order to fully leverage them.
So what is a robust data strategy? A data strategy requires a plan for sourcing and storing data, ideas for putting data to work, and perhaps most importantly, a process for evaluating which ideas are and are not effective. Companies that successfully implement a robust strategy, and that continually evaluate and evolve their data use, can reap the rewards. Let’s take a look at what that means in practice.
Build A Data Infrastructure
In the world of machine learning and AI, you’re only as good as your data, so gathering as much high-quality data as possible is extremely important. Data is essential to training new models, evaluating them once deployed, and contextualising them within your larger business goals. In a robust data strategy, you’ll need to collect as much data as possible for as long as possible.
Fortunately, this is where the cloud excels. Storing data in the cloud is extremely easy and very cheap. For most organisations, the main constraint is not technological but legal. Many organisations choose to store as much as possible under the GDPR, or whatever the prevailing privacy regulation is. When designing a strategy, you should understand the legal and budgetary concerns here, but focus on storing as much as permissible.
After that, the main concern is how to store data. Most cloud providers have great support for storing flat files (think Amazon’s S3 or Azure’s Blob Storage), which is cheap and fast. Even better than flat files, however, is storing data in a database. This structured storage will be incredibly useful down the road, and you’ll benefit from the database format with faster development times and more accurate models down the road. The cloud has a lot to offer here as well, from managed single database solutions to big data tools like Google’s BigTable.
One other area where the cloud shines here is in data management. With privacy and security at the forefront of everyone’s mind these days, implementing automated policies for data management can save a lot of hassle. With new tools for expiring data automatically and auditing access, compliance comes built in.
Build An Insights Infrastructure
Once you’ve got your data, it’s time to put it to work. In the past 5 years, we’ve seen an explosion of cloud providers offering sophisticated, powerful tools for gleaning insights from well-structured data. So where to begin?
Start with the basics, like AWS Sagemaker or similar offerings from other providers. These tools give you access to the same machine learning algorithms Amazon and Google are using internally, but in a controlled environment. The goal here is to get a firm understanding of what’s possible. See what business problems you could be solving, and make sure your expectations are aligned with what the framework provides. Remember, even simple-seeming models can provide massive improvements on old, static code.
Once you’ve gotten the hang of it, the next place to turn is the AI or Insights Hub. Because AI is so essential to cloud operations, every cloud provider has a well-designed portal for seeing what models can be trained, what data you have to work with, and what the accuracy and other top-level numbers are. AI and insights are a rapidly-evolving area in the cloud landscape, so it’s good to check-in periodically and see what’s changed.
Finally, consider building out your own insights architecture. More and more providers are offering specialised machine learning servers, and it’s easier and more cost-effective than ever to train your own models. Cutting edge algorithms are also typically open source, and can be deployed on regular cloud instances. The state of the art is truly within your grasp.
A data strategy is not a single process, it’s a continual process for bettering how you deal with data. It’s not uncommon to deploy a production machine learning model and immediately start collecting new data for the next version, with the knowledge of what was missing from the first. The final step in any data strategy is to evaluate and assess how your data operations are going, and improve.
One great feature of the cloud is transparent billing. As you consume APIs for insights, and structured data storage, you can directly see the costs and the business value being produced. This is extremely useful for assessing model costs, and see the full supply chain of value, from the data input to the model, to the cost of training it, to the cost for every API call.
The other big win from the cloud comes from automation. Once you’ve developed your first AI model you have a blueprint for doing it again and again. Not only do you have the developer experience, you also have the ability to “copy and paste” the component resources in the cloud, both the data that went into training the model to the servers it was deployed on. You can find what works and duplicate it endlessly.
Developing a robust data strategy is difficult but the payoff can be enormous. When properly implemented, an insights pipeline creates enormous value for your enterprise. We’ve got expertise in developing targeted data strategies that generate impactful, actionable, and valuable insights. Reach out today for a consultation, strategy session, or help implementing your own strategy.
A new methodology, the Flow methodology, is taking the lessons learned from Agile and putting them...Read More
In this article, we’ll take a look at a few ways to optimise your cloud spend. Even if your monthl...Read More