Run Commands

S3 Gateway

Learn about the embedded S3 gateway, which is compatible with MinIO, AWS S3 CLI, and boto3.

May 30, 2023

Use the embedded S3 Gateway to send or receive data through the S3 protocol using object storage tooling such as Minio, boto3, or AWS s3 CLI. Operations available are similar to those officially documented for S3.

S3 Gateway Syntax #

The S3 gateway presents each branch from every Pachyderm repository as an S3 bucket. Buckets are represented via [<commit>.]<branch>.<repo>.<project>, with the commit being optional.

If auth is enabled, credentials must be passed with each S3 gateway endpoint as mentioned in S3 Client configuration.

Command Examples #

The following command examples assume that you have upgraded to use the embedded proxy, which will become mandatory in future releases.

Put Data Into Pachyderm Repo #

Tool:

Retrieve Data From Pachyderm Repo #

Tool:

Port Forwarding #

You can pachctl port-forward to access the s3 gateway through the localhost:30600 endpoint, however, the Kubernetes port forwarder incurs substantial overhead and does not recover well from broken connections.