Skip to content

Pachyderm 2.3.5 is out! Download here. 2.3.0 and later versions support ARM-based architectures!!!

Pachyderm Documentation Overview

pachyderm/pachyderm

Home
Getting Started
Concepts
How-Tos
Deploy and Manage
Reference
Enterprise
Troubleshooting

Pachyderm Documentation

pachyderm/pachyderm

Home
Home
- Welcome
- Contributing
  Contributing
Getting Started
Getting Started
Concepts
Concepts
- Overview
- Versioned Data Concepts
  Versioned Data Concepts
  - Overview
  - Repository
  - Commit
  - Branch
  - File
  - Provenance
  - History
- Pipeline Concepts
  Pipeline Concepts
  - Overview
  - Pipeline
    Pipeline
    
    Overview
    
    Cron
    
    Service
    
    Spout
  - Job
  - Datum
    Datum
    
    Overview
    
    Pipeline Inputs
    Pipeline Inputs
    
    Glob Patterns
    
    Cross and Union
    
    Join
    
    Group
    
    Datum Processing
    
    Datum Processing States
    
    Datum Metadata
- Advanced Concepts
  Advanced Concepts
How-Tos
How-Tos
- Overview
- Workflow
  Workflow
- Basic Data Operations
  Basic Data Operations
  - Overview
  - Ingest Your Data Into Pachyderm
    Ingest Your Data Into Pachyderm
    
    Load Data with pachctl
    
    Use the SQL Ingest Tool
  - Export Your Data From Pachyderm
    Export Your Data From Pachyderm
    
    Export Your Data with pachctl
    
    Export Your Data with egress
    
    Egress to an SQL Database
    
    Mount a Repo to a Local Computer
  - Delete a Commit / Delete Data
- Pipeline Operations
  Pipeline Operations
- Advanced Data Operations
  Advanced Data Operations
- Use Console
- Use JupyterLab Mount Extension
Deploy and Manage
Deploy and Manage
- Overview
- Deploy Pachyderm
  Deploy Pachyderm
  - Overview
  - Helm Deployment Principles
  - Local Deployment
  - QUICK Cloud Deployment
  - NEW - Deploy Pachyderm With a Proxy
  - Production Deployment
    Production Deployment
    
    Load Balancer and Ingress Setup
    Load Balancer and Ingress Setup
    
    Infrastructure Recommendations
    
    Set Up Traefik
    
    Set Up Secrets Manager
    Set Up Secrets Manager
    
    AWS Secrets Manager
    
    Deploy on AWS
    
    Deploy on GCP
    
    Deploy on Azure
    
    Deploy on premises
  - Deploy Console
  - Deploy JupyterLab Extension
  - Additional Customizations
    Additional Customizations
    
    Import a Kubernetes Context
    
    Deploy Pachyderm with TLS
    
    Deploy in a Custom Namespace
    
    Configure RBAC
  - Post-Deployment
    Post-Deployment
    
    Connect to a Pachyderm cluster
    
    Configure Tracing with Jaeger
    
    Enable logs aggregation with Loki
    
    Monitor cluster metrics with Prometheus
    Monitor cluster metrics with Prometheus
    
    Set up Prometheus
    
    Configure Environment Variables
    Configure Environment Variables
    
    Overview
    
    Enterprise
    
    Global
    
    OS
    
    Pachd
    
    Storage
    
    Workers
- Manage Pachyderm
  Manage Pachyderm
  - Overview
  - Manage Cluster Access
  - Use the S3 Gateway
    Use the S3 Gateway
    
    Global S3 Gateway
    
    Configure your S3 Client
    
    Sidecar S3 Gateway
    
    Supported Operations
    
    Unsupported Operations
  - Disable Usage Metrics
  - Upgrades and Migrations
    Upgrades and Migrations
    
    Overview
    
    Upgrade your Cluster
  - Backup and Restore
  - Storage Use and GPUs
    Storage Use and GPUs
    
    Storage Use Optimization
    
    Use GPUs
Reference
Reference
Enterprise
Enterprise
- Overview
- Features
- Deploy Enterprise Edition
- Authentication & Authorization
  Authentication & Authorization
  - Activation
  - Authentication
    Authentication
    
    Connect your IdP
    
    Login Flow
  - Authorization
    Authorization
    
    Model overview
    
    Role Binding
    
    Permissions
- Enterprise Server
  Enterprise Server
  - Setup an Enterprise Server
  - Manage your Enterprise Server
Troubleshooting
Troubleshooting

How-Tos¶

This section includes how-tos that describe best practices of data operations in Pachyderm.

Workflow

Organize your Pachyderm workflow.

Developer Workflow
Working with Data and Pipelines
Tag and Push your Image Flag
CI/CD Integration
Create a Machine Learning Workflow

Basic Data Operations

Ingest your data into Pachyderm and export them out.

Ingest/Export Strategies Overview
Delete Data

Pipeline Operations

Learn pipeline operations.

Test datums
Create a Pipeline
Update a Pipeline
Delete a Pipeline
Use Jsonnet Pipeline Specs

Advanced Data Operations

Manage Secrets, use transactions, skip failed datum...

Create and Manage Secrets
Processing Time-Windowed Data
Use Transactions
Skip Failed Datums
Attach a Volume to your Pipeline

Other

Pachyderm JupyterLab Extension

Use Pachyderm Mount Extension

Last update: July 23, 2022

Does this page need fixing?

Previous Distributed Computing