Platform Overview

The Fastest Way to Build Pipelines for
Unstructured Data

The end-to-end platform to work with unstructured data. Get live in minutes with pre-built components, or run custom pipelines on our fully managed infrastructure to save months of engineering time.

See Pipeline Creation in Action

High quality data sources

We offer fully-managed connectors from industry leading providers. Tap into user-generated content at scale with agile data procurement.

Extract Insights from Anywhere

Integrated

Data sources that are ready out-of-the-box. Simply select them in your queries.

Adapter

Fully managed connectors that only require your API key/token to use.

Your Own Data

Pull internal data through an API or by dragging & dropping documents.

Automated schema standardization

Say goodbye to manual data indexing and laborious schema unification. Free engineers from data plumbing to speed up product development.

Transform unstructured data to be:

Create Robust Data Pipelines in Minutes

Example Pipeline – Consumer Insights: This data pipeline feeds enriched user-generated content into a consumer insights product used by Fortune 500 companies.

Easily deploy AI & NLP operations

Our tool is not just about ingesting vast quantities of data, it’s about making sense of it. Filter out the noise and enrich data with built-in AI models developed by world class teams.

AI Enrichments with a Simple Query

Example Pipeline: Search Twitter & Instagram for posts with the word “cats”, and then enrich results with sentiment & intent classifiers.

API Query

				
					curl --location 'https://api.platform.datastreamer.io/api/search' \
--header 'apikey: xxx' \
--header 'Content-Type: application/json' \
--data '{
    "query": {
        "query": "cats",
        "data_sources": [
            "wsl_twitter",
            "wsl_instagram"
        ],
    },
    "operations": [
        {
            "name": "sentiment",
            "destination_path": "operations.sentiment",
            "parameters": {
                "language": "enrichment.language",
                "main": "content.body"
            }
        },
        {
            "name": "intent",
            "destination_path": "operations.intent",
            "parameters": {
                "language": "enrichment.language",
                "main": "content.body"
            }
        }

Sample Result

				
					{
    "results": [
        {
            "id": "1681502878992200182-artemis",
            "data_source": "wsl_twitter",
            "source": {
                "link": "https://twitter.com/Uncanny_Vanny/status/xxxx"
            },
            "content": {
                "body": "Honeslty love the idea that William is a cat person. Like yeah he loves bunny themed stuff but cats would be a close second.",
                "found": "2023-04-14T20:07:58Z",
                "published": "2023-04-14T20:07:15Z",
                "favorites": 0,
                "followers": 163,
                "following": 0
            },
            "author": {
                "name": "🐇🔪💜 VANNY 💜🔪🐇 #FireLadyFiszi",
                "bio": "🐇🔪💜 -VANNY- 💜🔪🐇",
                "location": "Freddy's Mega Pizzaplex",
                "profile_image_source": "https://pbs.twimg.com/profile_images/xxxx/fbvJisCU_normal.jpg",
                "gender": "UNKNOWN",
                "url": "https://twitter.com/xxxx",
                "handle": "xxxx"
            },
            "enrichment": {
                "language": "en",
                "sentiment": "POSITIVE",
                ]
            },
            "twitter": {
                "tweet_type": "POST",
                "retweet_type": "QUOTE",
                "post_identifier": "1646968378366763009",
                "user_verified": false,
                "user_id": "1542407216115830784"
            },
            "operations": {
                "sentiment": {
                    "label": "positive",
                    "confidence": 0.9996
                },
                "intent": {
                    "label": "yes",
                    "confidence": 0.9889
                }
                    ]
                }
            }
        }
    ],
    "total": {
        "value": 10000,
        "relation": "gte"
    }
}

Integrate Seamlessly Into Your Data Stack

Move data where you need it

No more compatibility issues with complex data – just swift and simple integration into your warehouses or infused into your products with zero hassle.

High-speed search storage

Enter a world where your data is not just stored, but optimally organized and ready for action. Our data lakes keep your data conveniently accessible for robust data pipelines.

Query unstructured data

Find the needle in the haystack without drowning in complexity. Filter data with simple queries to boost analytical productivity.

Monitor web data for insights

Cater to your customers’ need for rapid insights from fluctuating data sources. Expand your coverage to billions of documents without the strenuous work.

The Fastest Way to Build Pipelines for Unstructured Data