Object & File Storage

MarcoPolo connects to object storage (AWS S3, Azure Blob Storage), cloud file storage (Google Drive, OneDrive, SharePoint), and SFTP servers. Your AI can browse directories, download files, and process data from any of them. Useful for log analysis, data lake exploration, document processing, and file-based workflows.

Connecting

Provide your storage credentials in the web UI. For S3, you provide bucket names and an AWS access key pair. For Azure Blob Storage, you provide container names, the storage account name, and Microsoft Entra service-principal credentials. For Google Drive, OneDrive, and SharePoint, you authenticate via OAuth (Google Drive also accepts a service account key). For SFTP, you provide host, port, username, private key, and root path.

Credentials are stored encrypted and never exposed to the AI.

Browse, download, query

Storage connections expose capability-gated verbs through the in-workspace connection CLI:

browse — list files at a provider-side path:

You: "Show me what's in the raw-logs bucket."
You: "List files in s3://data-lake/bronze/orders/ from last week."
You: "What spreadsheets are in my Google Drive?"

The AI runs connection browse <name> --remote-path ... --json through workspace_shell.

download — pull files into the workspace:

You: "Download the latest sales report from S3."
You: "Grab the Feb 11 partition from the orders data lake."
You: "Pull the Q1 revenue spreadsheet from OneDrive."

The AI runs connection download <name> --remote-path ... --json. Downloaded files land in data/downloads/ by default. You can override with --local-path if you want them grouped with a specific analysis.

query over downloaded files — DuckDB reads CSV, Parquet, JSON, and Excel directly from a workspace path:

You: "Parse this CSV, clean up the date columns, and load it into DuckDB."
You: "Read the JSON logs and count error types by hour."

The AI writes a query under connections/DUCKDB/queries/ that calls read_csv_auto('data/downloads/<file>.csv') (or read_parquet, read_json_auto, etc.) and runs it through connection query DUCKDB.

upload — round-trip transformed files back to the provider:

You: "Push the cleaned dataset back to S3."

The AI runs connection upload <name> --local-path ... --remote-path ... --json. upload is capability-gated — only available where the platform supports writes for that connection type. Each connection's capabilities are listed in connections/<name>/README.md and in connection list --json.

Common workflows

Log analysis. Browse log partitions, download a time range, parse and aggregate in DuckDB. Ideal for incident investigation: "The orders API has been slow since yesterday. Check the S3 event logs." See Triage & Break-Fix.

Data lake exploration. Navigate bronze/silver/gold layers, check partition sizes, sample files to understand structure. Answer questions like "Is this partition larger than normal?" or "What schema does this Parquet file use?"

File-based ETL. Download from storage, transform in DuckDB or with Python in scripts/, write outputs into artifacts/, optionally upload back. The workspace is a natural staging area for this kind of work. See Pipelines.

Best practices

Be specific about paths and time ranges. Storage buckets can contain millions of files. Give your AI a starting path or date range to narrow the search.

Sample before loading. For large files, ask your AI to check the file size first or download a sample. Loading a 10GB file into DuckDB is a different operation than loading a 10MB file.

Document storage structure in connections/<name>/RULES.md. If your data lake follows a specific layout (partitioned by date, organized by domain) or your Drive has a folder convention, document it so your AI knows where to look. See Context (RULES.md).