Skip to content

Pachyderm 1.13.4 is out! Download here.

Try on Hub! Free

Pachyderm Documentation Overview

pachyderm/pachyderm

Home
Getting Started
Concepts
How-Tos
Deploy and Manage
Reference
Enterprise
Troubleshooting

Pachyderm Documentation

pachyderm/pachyderm

Home
Home
- Welcome
- Contributing
  Contributing
Getting Started
Getting Started
Concepts
Concepts
- Overview
- Versioned Data Concepts
  Versioned Data Concepts
  - Overview
  - Repository
  - Commit
  - Branch
  - File
  - Provenance
  - History
- Pipeline Concepts
  Pipeline Concepts
  - Overview
  - Pipeline
    Pipeline
    
    Overview
    
    Cron
    
    Service
    
    Spout
  - Job
  - Datum
    Datum
    
    Overview
    
    Glob Patterns
    
    Datum Processing
    
    Cross and Union
    
    Join
    
    Group
    
    Datum Processing States
- Advanced Concepts
  Advanced Concepts
  - Deferred Processing of Data
  - Distributed Computing
How-Tos
How-Tos
- Overview
- Workflow
  Workflow
- Basic Data Operations
  Basic Data Operations
  - Overview
  - Ingest Your Data with pachctl
  - Export Your Data From Pachyderm
    Export Your Data From Pachyderm
    
    Export Your Data with pachctl
    
    Export Your Data with egress
    
    Mount a Repo to a Local Computer
  - Ingress and Egress Data from an External Object Store
  - Delete Data
- Pipeline Operations
  Pipeline Operations
- Advanced Data Operations
  Advanced Data Operations
- Use Pachyderm IDE
  Use Pachyderm IDE
  - Overview
  - Using the Pachyderm IDE with python-pachyderm
Deploy and Manage
Deploy and Manage
- Overview
- Deploy Pachyderm
  Deploy Pachyderm
  - Overview
  - Getting Started with Pachyderm Hub
  - Deploy with Helm
  - Production Deployments
    Production Deployments
    
    Deploy on GKE
    
    Deploy on AWS
    Deploy on AWS
    
    Overview
    
    Deploy Pachyderm on Amazon EKS
    
    Deploy Kubernetes with kops
    
    Deploy Pachyderm on AWS
    
    Deploy a Pachyderm Cluster with CloudFront
    
    Deploy on Azure
    
    Deploy on IBM CLoud
    
    Deploy on OpenShift
    
    Deploy On-Premises
  - Deploy Pachyderm IDE
  - Create a Custom Deployment
    Create a Custom Deployment
    
    Overview
    
    Before You Begin
    
    Pachyderm Deployment Manifest
    
    Configuring Persistent Disk Parameters
    
    Configuring Object Store
    
    Create a Complete Configuration
    
    Additional flags
  - Additional Customizations
    Additional Customizations
    
    Import a Kubernetes Context
    
    Deploy Pachyderm with TLS
    
    Deploy in a Custom Namespace
    
    Deploy a Custom Object Store
    
    Configure RBAC
  - Post-Deployment
    Post-Deployment
    
    Connect to a Pachyderm cluster
    
    Configure Ingress
    Configure Ingress
    
    Overview
    
    Pachyderm UI (Dash) Ingress
    
    Configure Tracing with Jaeger
    
    Enable logs aggregation with Loki
    
    Monitor cluster metrics with Prometheus
    Monitor cluster metrics with Prometheus
    
    Set up Prometheus
    
    List of pachd metrics
    
    List of pipeline metrics
    
    Configure Environment Variables
- Manage Pachyderm
  Manage Pachyderm
  - Overview
  - Manage Cluster Access
  - Use the S3 Gateway
    Use the S3 Gateway
    
    Global S3 Gateway
    
    Configure your S3 Client
    
    Sidecar S3 Gateway
    
    Supported Operations
    
    Unsupported Operations
  - Disable Usage Metrics
  - Upgrades and Migrations
    Upgrades and Migrations
    
    Overview
    
    Migrate to a Major Version
    
    Upgrade your Cluster
  - Backup and Restore
  - Optimize Performance
    Optimize Performance
    
    Storage Use Optimization
    
    Use GPUs
    
    Share GPU Resources
Reference
Reference
Enterprise
Enterprise
- Overview
- Features
- Deploy Enterprise Edition
- Configure Access Controls
  Configure Access Controls
  - Overview
  - Enable Access Controls
  - Manage Users and Groups
  - Manage Authentication Config
  - Configure a SAML Provider
    Configure a SAML Provider
    
    Configure SAML
    
    Use SAML
  - Configure an OIDC Provider
    Configure an OIDC Provider
    
    Overview
    
    Configure Pachyderm with Okta
    
    Configure Pachyderm with Keycloak
    
    Configure Pachyderm with Auth0
    
    Configure Pachyderm with Google OAuth 2.0
- Advanced Statistics
Troubleshooting
Troubleshooting

How-Tos¶

This section includes how-tos that describe best practices of data operations in Pachyderm.

Workflow

Organize your Pachyderm workflow.

Developer Workflow
Working with Data and Pipelines
Build pipelines
Build Flag
CI/CD Integration
Create a Machine Learning Workflow

Basic Data Operations

Ingest your data into Pachyderm and export them out.

Ingest/Export Strategies Overview
Ingress/Egress Data from an External Object Store
Delete Data

Pipeline Operations

Learn pipeline operations.

Test datums
Create a Pipeline
Update a Pipeline
Run a Pipeline on a Specific Commit
Delete a Pipeline
Monitor Job Progress

Advanced Data Operations

Manage Secrets, split your data, use transactions, skip failed datum...

Create and Manage Secrets
Split Data
Processing Time-Windowed Data
Use Transactions
Skip Failed Datums
Attach a Volume to your Pipeline

Last update: June 2, 2021

Does this page need fixing?

Previous Distributed Computing