Skip to content

New Features and Major Changes

Pachyderm 2.1, comes with new features and performance improvements.

On this page, we highlight Pachyderm 2.1 notable changes and point to their relevant documentation.

So, what is new?

New Common Core Feature

Jsonnet Pipeline Specifications

A jsonnet pipeline specification file is a wrapping function atop a JSON file, allowing you to parameterize pipeline specifications, thus adding a dynamic component to the creation and update of pipelines.

Visit the Jsonnet pipeline specifications page to learn more about our parameterized pipeline specs.

Additionally, once you are familiar with the opencv, group or the join examples, we have added a Jsonnet version of their pipeline specs (opencv jsonnet pipeline specs, retail group jsonnet pipeline spec, outer join jsonnet pipeline spec) as an illustration of how they can be used.

More Change

More useful changes:

  • We are now exposing the capture groups of the JOIN_ON and GROUP_BY to the pipelines jobs in environment variables:

    • The content of the capture group defined in the group_by parameter is available to your pipeline's code in the environment variable: PACH_DATUM_<>_GROUP_BY.
    • Similarly, the content of the capture group defined in the join_on parameter is available to your pipeline's code in the environment variable: PACH_DATUM_<>_JOIN_ON. This new addition will tremendously simplify your pipeline code. For illustrative purposes, we have used it in our examples.
  • Pachyderm now ships with a embedded Loki for a more persistent log collection.

New Enterprise Features

Console Improvements

  • We are introducing a new Getting Started Tutorial built right into Console to teach users about Pachyderm's key concepts, such as Data Versioning and Pipelines.
  • Drag-and-Drop File Upload: Console now supports file ingress from your local computer via an intuitive UI.
  • List View: Along with your traditional DAG view, our new List View provides an alternative to the DAG view that makes it easier to search and filter through all your repos and pipelines.

See Also

Check our Changelog.

Last update: August 5, 2022
Does this page need fixing? Edit me on GitHub