- Feature: Users can now manage metadata (as key:value pairs) in Console for projects and repositories from the User Metadata tab of the details side panel.
- Enhancement: Projects, pipelines, branches, and commits now include the following dervied metadata by default:
created_at
,created_by
,updated_at
, andupdated_by
. - Feature: Pre-built Jsonnet templates are now available in Console when creating a pipeline:
- Snowflake Integration: Creates a cron pipeline that can execute a query against a Snowflake database and return the results in a single output file.
- Hugging Face Downloader: Creates a cron pipeline to download datasets or models from huggingface on demand.
- Enhancement: Several enhancements have been made to improve the integration between Pachyderm and .
- Feature: The Pachyderm SDK now has an extras package
cdr
that you can install (pip install pachyderm_sdk[cdr]
) to make use of Common Data Refs (CDRs) in your user code. CDRs improve performance and speed by downloading version-controlled data directly from Pachyderm’s underlying Object Storage bucket and caching that data locally on your machine, allowing datasets to be assembled entirely locally and incrementally updated. - Security: The Pachyderm repository is now available at Iron Bank, a hardened container image repository owned and maintained by the U.S. Department of Defense (DoD) that supports the end-to-end lifecycle for modern software development. If you plan to download and install from Iron Bank, please reach out to
ai-support@hpe.com
or your Customer Success Engineer for assistance. - Notice: The
gocdk_enabled
attribute has been removed from the Helm Chart Values as it is now the default object storage driver.
Release Notes Highlights for Pachyderm
2.11.0
August 6, 2024
2.10.6
July 5, 2024
- Feature: You can now create pipelines in Console using jsonnet-compatible pipeline templates.
- Enhancement: Console now displays more information in the side panel for repos and pipelines, including the pipeline’s active workers and parallelism spec.
- Enhancement: Several web-responsive improvements have been made to Console: the side-panel now supports responsive sizing, and the DAG view side panel tab headers become scrollable on small screen sizes.
- Fix: Action buttons in the sidebar now become dropdown ellipsis menus when resizing the sidebar to a narrower size.
- Fix: When trying to download a single file, the file browser will download the file instead of opening the file in a new tab
- Fix: Correctly configured custom TLS certificates in Console.
- Fix: Improved our consistent hashing algorithm’s use of locking.
2.10.4
June 18, 2024
- Enhancement: Editing Project and Cluster Metadata is now available in Console.
- Enhancement: pachctl is now available in Determined notebooks.
- Fix: Resolve an unexpected file loading error when opening multiple files in the Jupyter extension explore view.
- Fix: Pachyderm Clients are now correctly loaded into Determined notebooks.
2.10.0
May 5, 2024
- Feature: For enterprise customers using MLDM (Pachyderm) & MLDE (Determined) in a combined cluster environment, the MLDE Notebooks now include support for the Jupyter Pachyderm extension by default.
- Feature: You can now set up and maintain metadata on your Pachyderm artifacts. This includes clusters, projects, repos, branches, and commits.
- Enhancement: The blog storage configuration attribute
GOCDK_ENABLED
is now set totrue
by default in your Helm chart values; in 2.11.0 the option to disable it will be removed. - Enhancement: The Console UI has undergone several improvements, including:
- Improved file browsing experience
- Improved DAG visualizations
- Interactive DAG edge highlighting
- Distinguishing colors and/or patterns based on pipeline types
- Enhancing the ability to understand more about connections by their edges (like joins)
- Visual indications of parallelism for pipelines when spec calls for it and when running
- Pipeline and repo table paging
2.9.5
April 24, 2024
- Enhancement: Colors in the Console’s file browser have been improved for accessibility.
- Enhancement: Console’s callback page now properly redirects to the original page after login instead of redirecting to the home page (
/
). - Enhancement: The visualization of the DAG in Console have been improved for better sizing and scaling with your browser window.
- Fix Resolved a bug where pipeline and job statuses were not correctly displaying.
2.7.8
April 10, 2024
- Enhancement: Updated to Go 1.22.2 and latest x/net.
- Enhancement: Upgraded to Envoy 1.27.4
2.8.7
April 10, 2024
- Enhancement: Updated to Go 1.22.2 and latest x/net.
- Enhancement: pinned Pachyderm-SDK versino to >=2.8.0<2.9.0 for the Jupyter Mount Extension.
- Enhancement: Upgraded to Envoy 1.27.4
2.9.4
April 10, 2024
- Enhancement: Updated to Go 1.22.2 and latest x/net.
- Enhancement: Upgraded to Envoy 1.27.4
- Fix: Resolved issue in Console where proxy port caused issues with downloading files.
- Feature: Added new args to debug dumps. You can review the full debug template by running
pachctl debug template
in the CLI.
Full Default Debug template #
Expand
2.9.3
March 25, 2024
- Enhancement: Refactored the Console’s Explore View.
- Enhancement: You can now adjust the Refecth Interval rate for Console in the Helm Chart.
- New Feature: You can now set roles on the Cluster level from within Console.