# A trend prediction company runs about 75M posts a month through one pipeline

About 16,000 queries a day, structured on one schema, feeding the models global brands rely on.

About 75M posts a month · About 16,000 queries every 24 hours · Several years in production

## At a glance

- **Customer:** Consumer trend prediction company
- **Sources:** Social and web, plus monthly historical backfills
- **Volume:** About 75M posts a month
- **Delivery:** Its own systems, on the Datastreamer schema
- **In production:** Several years, fully automated

## About the customer

A consumer trend prediction company that tells global brands what people will want next.

**Industry:** Consumer and Market Intelligence

## The Story

Challenge, solution, results.

### The Challenge: Predictions with gaps in the data

The company tells global consumer brands what people will want next. Its models are only as good as the social data behind them, and that data had gaps: sources it couldn't reach, and data that arrived in different shapes depending on where it came from.

Every new source meant more collection code to write and keep running. That work sat on top of the modeling its customers actually pay for.

### The Solution: From one-off pulls to a data backbone

It started small, with one-off pulls to cover the sources its models were missing. Those pulls became scheduled runs. Today about 16,000 queries go out every 24 hours, across several languages, with no one kicking them off. Monthly historical backfills fill in the past.

The company already had its own data systems, built in-house. It adapted them to run on the Datastreamer schema, so every source lands in the same shape. Enrichment and structuring now happen inside the pipeline, before the data reaches its models.

### The Results: More time on predictions

About 75M posts a month now flow through one pipeline, cleaned, enriched and structured. Adding a source means adding a component to the pipeline. What began as a way to fill gaps is the data layer the business runs on, and its data team spends its time on predictions.

## How the pipeline works

1. **Sources:** Social and web sources, plus monthly historical backfills.
2. **Collection:** About 16,000 scheduled queries every 24 hours, in several languages.
3. **Processing:** Enrichment and structuring onto the Datastreamer schema.
4. **Delivery:** Into the company's own systems and models.

## Ready to build your data layer?

Bring the sources you're missing into one pipeline, structured the way your models need them.

[Talk to Sales](https://datastreamer.io/book-demo/) · [Read the Documentation](https://docs.datastreamer.io)

## Related

- [Risk Intelligence Case Study](https://datastreamer.io/risk-intelligence-case-study/)
- [Listening Tool Replacement Case Study](https://datastreamer.io/listening-tool-replacement-case-study/)
- [External Data Ingestion for Market Intelligence Platforms](https://datastreamer.io/external-data-ingestion-for-market-intelligence-platforms/)