Data pipeline recipe
Automatically analyze daily brand sentiment for thousands of brands on Tiktok.
Recipe Overview
If your platform helps brands analyze sentiment, measure brand health, or surface early signals, then unstructured data from Tiktok is gold!
Building reliable pipelines to collect that data?
That’s a mess of APIs, scrapers, proxies, normalization logic, and edge-case cleanup.
This recipe gives your team a pre-configured Datastreamer pipeline to ingest and process that data. This pipeline pulls from reliable open sources, enriches the content, and outputs structured results ready for downstream use.
The pipeline created by this recipe will automatically pull data from a Tiktok aggregator, normalize the data to a common schema, and then deliver into a workflow of NLP and LLM-based enrichments. These enrichments have been ordered and filtered to reduce running costs for the customer. Delivery to a webhook is listed, but is one of the many delivery options.
- Goal:
Ingest and normalize mentions of tracked keywords (brand or competitor names) across Tiktok, enrich with sentiment, and deliver into your platform.
- Deployment Time:
~5 minutes.
- Output Format
JSON records delivered as a webhook.
- Pipeline Components
- Tiktok data aggregator (Ingress)
- Schema transformation (Transformation)
- JSON normalization (Transformation)
- NLP Entity extraction (Enrichment)
- AI Slang translation (Enrichment)
- AI Sentiment extraction (Enrichment)
- Webhook (Egress)
Setup Instructions
- 1. Import the Pipeline File
- 2. Configure & Automate Search Criteria
Create Periodic Jobs using keywords, hashtags, or other search criteria. Using Periodic Jobs will automate new data ingestion.
- 3. Configure Your Delivery
In this recipe, JSON records are delivered using webhook. (Optional: Snowflake, Elastic, Cloud Storage, or other delivery)
- Describe & Deploy
Name your new pipeline, add descriptions, and deploy! Orchestrator will handle all the initiation of the various steps.
Need More Details?
View this Pipeline Configuration in our documentation Recipes for step-by-step explanation of each part.
Post-deployment, ready for product integration.
With your newly deployed pipeline, you can:
- Index this into your search UX or frontend dashboards
- Deliver to end customers as a stream of signals for brand intelligence
- Feed it into alerts and enrichment pipelines.
And because your pipeline is built on Datastreamer, it’s:
- Modular
- Scalable
- Compatible with your existing architecture
Don’t stop now.
Tweak, modify, and add to this recipe using over 38,000+ different capabilities.
Already using Datastreamer?
Not yet using Datastreamer?
FAQ’s
Yes, you can tweak and add any element in the Pipelines you create with any recipe. These configuration files simply lay out starting workflows and are fully customizable in your enviroment.
Yes, anything is possible. Datastreamer has pre-built integrations for ingesting data from social platforms, review sites, forums, trend tools, or your own feeds. Datastreamer handles structured and unstructured inputs.
Yes. Datastreamer has a full registry of ready-to-use NLP models. You can also use LLMs in pipelines or plug in your own logic and systems.
Yes. You can define enrichment rules, use your taxonomies, or plug in existing models.
Yes. Many enterprises use Datastreamer to power custom dashboards or workflows. Your pipelines can handle the ingestion, enrichment, and routing, so your team can focus on the interface and outcomes.
We abstract each source into a common schema, so your downstream systems don’t have to handle structural drift, missing fields, or format mismatches.
No problem. We support branching pipelines, so the same enriched data can be routed to multiple destinations: dashboards, warehouses, alerting systems, and more.
Yes, your new Pipelines can support any form of data. Your own data sources, 3rd party feeds, and social listening platforms can be used together.
Faster time to market, no ongoing maintenance, and a much lower risk surface for managing content ingestion and normalization.
