Understanding Datasets¶

Datasets are the core organizational unit in Gaia. A dataset is a named collection of indexed data from one or more Cohesity-managed sources. Every query you send to Gaia runs against one or more datasets.

What is a Dataset?¶

Think of a dataset as a searchable index over a slice of your enterprise data. When you create a dataset, you tell Gaia:

• Which data sources to include (backup snapshots, file shares, M365 mailboxes, etc.)
• A name to reference it by in API calls

Gaia then indexes the content — extracting text, chunking it, generating embeddings, and storing everything in its vector store. Once indexing is complete, the dataset is ready for queries.

┌─────────────────────┐
│    Data Sources      │       ┌───────────────────────┐
│                      │       │       Dataset          │
│  • Backup snapshots  │──────▶│  "financial-reports"   │
│  • File shares       │       │                        │
│  • M365 mailboxes    │       │  Status: ready         │
│  • Uploaded files     │       │  Objects: 1,247        │
│                      │       │  Last indexed: 2h ago  │
└─────────────────────┘       └───────────────────────┘
                                        │
                                        ▼
                               ┌────────────────┐
                               │  Vector Store   │
                               │  (Embeddings)   │
                               └────────────────┘

Dataset Lifecycle¶

Every dataset goes through the following stages:

Create  ──▶  Configure Sources  ──▶  Index  ──▶  Query
                                       ▲            │
                                       │            │
                                       └── Re-index ┘

1. Create¶

A new, empty dataset is created with a name and optional description.

2. Configure Sources¶

Data sources are attached to the dataset — these define what data Gaia should index.

3. Index¶

Gaia processes the data sources: extracts text from documents, splits content into chunks, generates vector embeddings, and populates the vector store. This can take minutes to hours depending on data volume.

4. Query¶

Once indexing completes, the dataset is in ready state and can be queried via /ask, /ask/stream, or /ask/exhaustive.

Re-index¶

When source data changes (new backups, updated files), you can trigger re-indexing to keep the dataset current.

Dataset States¶

Types of Data Sources¶

Gaia can index data from any source that Cohesity manages:

API Operations¶

List All Datasets¶

Retrieve a list of all available datasets.

async with GaiaClient.from_env() as gaia:
    datasets = await gaia.list_datasets()

    for ds in datasets:
        print(f"{ds.name:30s}  status={ds.status}  objects={ds.object_count}")

Response fields on each dataset:

Get Dataset Details¶

Retrieve detailed information about a specific dataset, including its data sources and indexing statistics.

async with GaiaClient.from_env() as gaia:
    details = await gaia.get_dataset("financial-reports")

    print(f"Name:    {details.name}")
    print(f"Status:  {details.status}")
    print(f"Objects: {details.object_count}")

    if details.data_sources:
        print("\nData Sources:")
        for src in details.data_sources:
            print(f"  • {src.get('type', 'unknown')}: {src.get('name', 'N/A')}")

    if details.indexing_stats:
        stats = details.indexing_stats
        print(f"\nIndexing Stats:")
        print(f"  Total docs:   {stats.get('totalDocuments', 'N/A')}")
        print(f"  Indexed docs: {stats.get('indexedDocuments', 'N/A')}")
        print(f"  Failed docs:  {stats.get('failedDocuments', 'N/A')}")

Create a Dataset¶

Create a new dataset programmatically.

async with GaiaClient.from_env() as gaia:
    result = await gaia.create_dataset(
        name="incident-logs-2025",
        description="Server and application logs from 2025 backups",
    )
    print(f"Created dataset: {result}")

Trigger Indexing¶

Start or restart the indexing process for a dataset.

async with GaiaClient.from_env() as gaia:
    result = await gaia.trigger_indexing("incident-logs-2025")
    print(f"Indexing triggered: {result}")

Delete a Dataset¶

Remove a dataset and its indexed data.

async with GaiaClient.from_env() as gaia:
    result = await gaia.delete_dataset("old-dataset")
    print(f"Deleted: {result}")

Monitoring Indexing Progress¶

For large datasets, indexing can take a significant amount of time. Here's how to monitor progress:

import asyncio
from gaia_sdk import GaiaClient


async def wait_for_indexing(dataset_name: str, poll_interval: int = 30):
    """Poll a dataset until indexing is complete."""
    async with GaiaClient.from_env() as gaia:
        while True:
            details = await gaia.get_dataset(dataset_name)

            if details.status == "ready":
                print(f"\n✓ Dataset '{dataset_name}' is ready!")
                if details.indexing_stats:
                    stats = details.indexing_stats
                    print(f"  Total documents:   {stats.get('totalDocuments', 'N/A')}")
                    print(f"  Indexed documents: {stats.get('indexedDocuments', 'N/A')}")
                return details

            if details.status == "error":
                print(f"\n✗ Dataset '{dataset_name}' encountered an error.")
                return details

            # Still indexing
            progress = ""
            if details.indexing_stats:
                stats = details.indexing_stats
                total = stats.get("totalDocuments", 0)
                indexed = stats.get("indexedDocuments", 0)
                if total > 0:
                    pct = (indexed / total) * 100
                    progress = f" ({indexed}/{total} = {pct:.0f}%)"

            print(f"  ⟳ Indexing{progress}... checking again in {poll_interval}s")
            await asyncio.sleep(poll_interval)


asyncio.run(wait_for_indexing("incident-logs-2025"))

Best Practices¶

Naming Conventions¶

Use clear, descriptive, lowercase names with hyphens:

Dataset Organization¶

• Group related data. Combine data sources that are commonly queried together into a single dataset.
• Separate by access control. If different user groups should access different data, use separate datasets and control access at the session/application level.
• Create temporal datasets. For compliance or forensics, create datasets scoped to specific time ranges (e.g., logs-q1-2025).

Refresh Schedules¶

• Static data (historical archives, compliance snapshots) — Index once, no refresh needed.
• Slowly changing data (policies, runbooks, documentation) — Re-index weekly or on change.
• Frequently changing data (logs, emails, tickets) — Re-index daily or trigger re-indexing when new backups complete.

Querying Multiple Datasets¶

You can query across multiple datasets in a single /ask call by passing multiple names:

async with GaiaClient.from_env() as gaia:
    response = await gaia.ask(
        dataset_names=["it-runbooks", "incident-logs-2025"],
        query="How do we handle a database failover?",
    )

Gaia merges results from all specified datasets and uses the combined context to generate the answer. This is powerful for cross-domain queries like "What was the root cause of the March 5 outage?" that might span infrastructure logs, runbooks, and incident tickets.

Next Steps¶

Now that you understand datasets, you're ready to build:

• Datasets & Indexing — Deep dive into creating, configuring, and indexing datasets.
• Querying & RAG — Learn how to query your datasets with the /ask endpoint.
• Backend with FastAPI — Wrap these operations in a proper backend.