Datasets & Indexing¶

Datasets are the foundational building block of every Gaia application. A dataset is a named collection of data sources — file shares, NAS volumes, object stores, or uploaded documents — whose contents are chunked, embedded, and stored in a vector index so Gaia can retrieve them at query time.

This page walks through the full lifecycle: create → configure → index → monitor → query.

Key Concepts¶

Listing Datasets¶

Before creating anything, check what already exists.

import asyncio
from gaia_sdk import GaiaClient

async def main():
    async with GaiaClient(api_key="YOUR_API_KEY") as gaia:
        datasets = await gaia.list_datasets()
        for ds in datasets:
            print(f"{ds.name}  status={ds.status}  objects={ds.object_count}")

asyncio.run(main())

curl -s \
  -H "apiKey: $GAIA_API_KEY" \
  "https://helios.cohesity.com/v2/mcm/gaia/datasets" | python -m json.tool

Creating a Dataset¶

A new dataset needs at minimum a name and one or more data sources.

async with GaiaClient(api_key="YOUR_API_KEY") as gaia:
    result = await gaia.create_dataset(
        name="engineering-docs",
        description="Internal engineering wiki and design documents",
        dataSources=[
            {
                "sourceType": "CohesityView",
                "viewName": "eng-wiki",
                "includePaths": ["/design-docs", "/runbooks"],
                "excludePaths": ["/archive"],
            }
        ],
    )
    print(result)

curl -s -X POST \
  -H "apiKey: $GAIA_API_KEY" \
  -H "Content-Type: application/json" \
  "https://helios.cohesity.com/v2/mcm/gaia/datasets" \
  -d '{
    "name": "engineering-docs",
    "description": "Internal engineering wiki and design documents",
    "dataSources": [
      {
        "sourceType": "CohesityView",
        "viewName": "eng-wiki",
        "includePaths": ["/design-docs", "/runbooks"],
        "excludePaths": ["/archive"]
      }
    ]
  }'

Data Source Options¶

Configuring Indexing¶

Indexing configuration controls what gets indexed and when.

File Filters¶

Restrict which files are ingested based on extension or size:

await gaia.create_dataset(
    name="legal-contracts",
    dataSources=[
        {
            "sourceType": "CohesityView",
            "viewName": "legal-share",
            "fileFilters": {
                "includeExtensions": [".pdf", ".docx"],
                "excludeExtensions": [".tmp", ".bak"],
                "maxFileSizeMB": 100,
            },
        }
    ],
)

Indexing Window¶

An indexing window limits processing to specific hours so production workloads remain unaffected:

{
  "indexingWindow": {
    "enabled": true,
    "startHour": 22,
    "endHour": 6,
    "timezone": "America/Los_Angeles"
  }
}

Triggering Indexing¶

After creating a dataset you must trigger indexing to start the initial ingestion. Subsequent re-index runs pick up only new and modified files (incremental updates).

async with GaiaClient(api_key="YOUR_API_KEY") as gaia:
    await gaia.trigger_indexing("engineering-docs")
    print("Indexing triggered.")

curl -s -X POST \
  -H "apiKey: $GAIA_API_KEY" \
  "https://helios.cohesity.com/v2/mcm/gaia/dataset/engineering-docs/index"

Monitoring Indexing Progress¶

Poll GET /dataset/{name}/details and inspect the indexingStats object to track progress.

async with GaiaClient(api_key="YOUR_API_KEY") as gaia:
    details = await gaia.get_dataset("engineering-docs")

    print(f"Status : {details.status}")
    print(f"Objects: {details.object_count}")

    if details.indexing_stats:
        stats = details.indexing_stats
        print(f"  Total files    : {stats.get('totalFiles', 'N/A')}")
        print(f"  Indexed files  : {stats.get('indexedFiles', 'N/A')}")
        print(f"  Failed files   : {stats.get('failedFiles', 'N/A')}")
        print(f"  Index size     : {stats.get('indexSizeBytes', 'N/A')} bytes")

curl -s \
  -H "apiKey: $GAIA_API_KEY" \
  "https://helios.cohesity.com/v2/mcm/gaia/dataset/engineering-docs/details" \
  | python -m json.tool

indexingStats Reference¶

Polling for Indexing Completion¶

For automation workflows you'll want to poll until indexing finishes:

import asyncio
from gaia_sdk import GaiaClient

async def wait_for_indexing(gaia: GaiaClient, name: str, poll_seconds: int = 10):
    """Poll until dataset indexing reaches a terminal state."""
    while True:
        details = await gaia.get_dataset(name)
        status = (details.indexing_stats or {}).get("status", "Unknown")
        indexed = (details.indexing_stats or {}).get("indexedFiles", 0)
        total   = (details.indexing_stats or {}).get("totalFiles", 0)

        print(f"[{name}] {status} — {indexed}/{total} files")

        if status in ("Completed", "Failed"):
            return details

        await asyncio.sleep(poll_seconds)

Re-indexing and Incremental Updates¶

Gaia supports incremental indexing by default. When you trigger indexing on an already-indexed dataset:

1. New files are chunked, embedded, and added to the index.
2. Modified files (detected via checksum or last-modified timestamp) are re-processed, and stale chunks are replaced.
3. Deleted files have their chunks removed from the index.

To force a full re-index (e.g., after changing chunking settings), pass the appropriate flag:

curl -s -X POST \
  -H "apiKey: $GAIA_API_KEY" \
  -H "Content-Type: application/json" \
  "https://helios.cohesity.com/v2/mcm/gaia/dataset/engineering-docs/index" \
  -d '{"fullReindex": true}'

Dataset Discovery¶

Once a dataset is indexed, the discovery endpoint provides a hierarchical view of all ingested content — useful for building file browsers or verifying that the right documents were picked up.

GET /dataset/{id}/discovery¶

async with GaiaClient(api_key="YOUR_API_KEY") as gaia:
    tree = await gaia.get_discovery("engineering-docs")

    def print_tree(node, indent=0):
        prefix = "  " * indent
        print(f"{prefix}{node.get('name', '/')}  ({node.get('type', 'dir')})")
        for child in node.get("children", []):
            print_tree(child, indent + 1)

    print_tree(tree)

curl -s \
  -H "apiKey: $GAIA_API_KEY" \
  "https://helios.cohesity.com/v2/mcm/gaia/dataset/engineering-docs/discovery" \
  | python -m json.tool

The response is a recursive tree structure:

{
  "name": "/",
  "type": "directory",
  "children": [
    {
      "name": "design-docs",
      "type": "directory",
      "children": [
        { "name": "architecture.pdf", "type": "file", "sizeBytes": 204800 },
        { "name": "api-spec.md", "type": "file", "sizeBytes": 15360 }
      ]
    }
  ]
}

Complete Example: Create → Index → Poll → Query¶

Putting it all together — a single script that creates a dataset, triggers indexing, waits for completion, and runs a test query.

import asyncio
from gaia_sdk import GaiaClient

DATASET_NAME = "quarterly-reports"

async def main():
    async with GaiaClient(api_key="YOUR_API_KEY") as gaia:

        # 1. Create the dataset
        print("Creating dataset …")
        await gaia.create_dataset(
            name=DATASET_NAME,
            description="FY25 quarterly earnings reports",
            dataSources=[
                {
                    "sourceType": "CohesityView",
                    "viewName": "finance-share",
                    "includePaths": ["/reports/FY25"],
                    "fileFilters": {
                        "includeExtensions": [".pdf", ".xlsx"],
                    },
                }
            ],
        )

        # 2. Trigger indexing
        print("Triggering indexing …")
        await gaia.trigger_indexing(DATASET_NAME)

        # 3. Poll until complete
        while True:
            details = await gaia.get_dataset(DATASET_NAME)
            stats = details.indexing_stats or {}
            status = stats.get("status", "Unknown")
            indexed = stats.get("indexedFiles", 0)
            total = stats.get("totalFiles", 0)

            print(f"  Indexing: {status} — {indexed}/{total} files")
            if status in ("Completed", "Failed"):
                break
            await asyncio.sleep(10)

        if status == "Failed":
            print("Indexing failed. Check the cluster logs.")
            return

        # 4. Run a test query
        print("\nRunning test query …")
        response = await gaia.ask(
            dataset_names=[DATASET_NAME],
            query="What was the Q3 revenue?",
        )
        print(f"Answer: {response.response_string}")

        if response.documents:
            print(f"\nSource documents ({len(response.documents)}):")
            for doc in response.documents:
                print(f"  • {doc.filename} (score: {doc.score:.3f})")

asyncio.run(main())

Best Practices¶

Platform Limits & Considerations¶

Supported File Types¶

Gaia can index the following file formats:

Indexing Modes¶

When creating a dataset, you choose an indexing mode that determines how and when data is processed:

Continuous Indexing¶

Gaia automatically checks for new snapshots every hour. When a new snapshot is available, indexing begins automatically.

• The dataset always reflects the latest data
• If a file is removed in a newer snapshot, it's no longer available in the dataset
• Topic Explorer topics refresh every 7 days
• Requires Cohesity cluster version 7.2.2_u1 or later

await gaia.create_dataset(
    name="live-reports",
    indexing_mode="continuous",
    dataSources=[{"sourceType": "CohesityView", "viewName": "reports"}],
)

One-Time Indexing¶

Index a specific point-in-time snapshot:

• Most recent snapshot — Indexes the latest available backup
• Custom date range — Indexes data within a specific time window (currently available for Microsoft 365 object types only)

On-Demand Indexing¶

For datasets created with Continuous Indexing, you can trigger an immediate index refresh:

curl -s -X POST \
  -H "apiKey: $GAIA_API_KEY" \
  "https://helios.cohesity.com/v2/mcm/gaia/dataset/live-reports/index"

Inclusion & Exclusion Rules¶

When creating datasets, you can filter what gets indexed using file types and directory paths.

File Type Filters¶

Include or exclude specific file extensions. By default all supported types are included.

Directory Path Rules¶

Use regex patterns to include or exclude specific directory paths:

What's Next¶

With datasets created and indexed, you're ready to start querying.

• Querying & RAG — learn how Gaia's retrieval-augmented generation pipeline works.
• Topic Explorer — explore themes and suggested questions from your indexed data.
• Document Upload — add ad-hoc files to a dataset via the upload API.
• Dataset Discovery — explore the indexed content hierarchy in detail.