Datastreamer, a data pipeline platform founded in December 2020, hit the limits of its infrastructure as it grew. The company processes massive volumes of unbounded data streams, including millions of news articles a day, and needed to scale beyond an on-premises setup. Working with Google Cloud and reseller partner SADA, the team migrated its pipeline to the cloud and reduced cloud costs by 30 percent compared to its previous on-premises operation.
The business challenge
Datastreamer's platform is built around data that never stops arriving. As Tyler Logtenberg, Vice President of Operations at Datastreamer, put it:
"Our primary focus is unbounded data streams that essentially don't have any limits. For example, there is no point at which you can say you have all the news. It just keeps coming."
Tyler Logtenberg, Vice President of Operations, Datastreamer
The platform runs searches across a petabyte of text data while applying enrichments in real time at very high volume. That scale is exactly what the company sells to its customers.
"With this unique platform, customers decrease their time to market and reduce data integration costs without the need for additional infrastructure, development, or maintenance."
Tyler Logtenberg, Vice President of Operations, Datastreamer
To keep delivering that at growing volumes, the company needed a foundation that could handle the speed and cost of its operations.
"We needed to scale for the future and migrate to a stable cloud infrastructure that could handle the speed and volumes of our operations, yet manage costs."
Tyler Logtenberg, Vice President of Operations, Datastreamer
The team was clear that keeping the platform stable and scalable should not be a distraction from the core product.
"Every second we are running over 10,000 enrichments from six different AI models in real time. When it comes to platform scalability and stability issues, that needs to be someone else's problem. We must be 100 percent focused on unbounded data streams."
Tyler Logtenberg, Vice President of Operations, Datastreamer
Choosing Google Cloud
Datastreamer selected Google Cloud for its next-generation platform. The decision came down to compute performance, container orchestration, and strong alignment with the technologies the company had already built in-house.
"There were a few reasons we chose to go all in on Google Cloud. First, Compute servers and Google Kubernetes Engine are far more advanced than the competition and accommodate our heavy use of Dataflow to run our models."
Tyler Logtenberg, Vice President of Operations, Datastreamer
"Also, Google Cloud's own products specialize in high volumes of searchable text data, which parallel a lot of the technologies that we've built in-house, so there is strong strategic alignment."
Tyler Logtenberg, Vice President of Operations, Datastreamer
The Google Cloud services central to the migration included:
- Google Compute Engine
- Google Kubernetes Engine (GKE)
- Dataflow, used to run the company's machine learning models
- Google Cloud products specializing in high volumes of searchable text data
Partnering with SADA
At Google Cloud's recommendation, Datastreamer partnered with SADA, a three-time Google Cloud Reseller Partner of the Year, for cost optimization and technical guidance. SADA became a fast, dependable source of answers as the team evaluated new technology.
"It's been great to go to SADA and say, 'We're doing a proof of concept for this new technology and we have three questions,' and then get quick answers. The research support and advice we receive from SADA is invaluable and keeps us from going down a rabbit hole."
Tyler Logtenberg, Vice President of Operations, Datastreamer
SADA also introduced Committed Use Discounts (CUDs), the primary mechanism behind the cost savings, making spending both lower and more predictable.
The results
The migration delivered measurable improvements across cost, stability, and capability:
- 30 percent reduction in cloud costs compared to on-premises operations
- Committed Use Discounts that made expenses predictable and lower
- Infrastructure that scaled to support rapid growth
- Improved service stability and performance
- New capabilities for secure customer data integration
"Compared to our previous on-premises situation, we reduced our cloud costs by 30 percent. Now, we are able to deliver more features, and because of the Committed Use Discounts that SADA showed us, our costs are more predictable and less expensive, a win-win in my book."
Tyler Logtenberg, Vice President of Operations, Datastreamer
On the new platform, Datastreamer now processes:
| Metric | Throughput |
|---|---|
| Real-time enrichments | 10,000 per second |
| Content pieces | 56,000 per second |
| Data points | 1.6 million per second |
Those enrichments run across six different AI models in real time, all against a petabyte of searchable text data.
Looking ahead
With the migration complete, Datastreamer plans to keep adopting new Google Cloud technology as it becomes available, with SADA continuing as a research and advisory partner.
"We're always pushing the limits on Google Cloud and keen to adopt its new technologies. SADA is invested in us finding those solutions that will enable our business and place us at a competitive advantage. They provide sage advice, bring in appropriate resources, and really go to bat for us."
Tyler Logtenberg, Vice President of Operations, Datastreamer
Datastreamer is a social and web data orchestration platform serving intelligence software companies.