Beginner Tutorial:
Explore Resources, Data, & Logs

Part 5: Explore Resources, Data, & Logs

A big part of building and maintaining a DAG of pipelines is understanding how to explore the resources, input/output data, and logs associated with your pipelines. Doing so will enable you to trouble-shoot issues, validate resource creation, and get a better understanding of how your pipelines are processing data.

List, Inspect, & Troubleshoot

Congratulations! You’ve successfully created a DAG of pipelines that process video files into a collage. However, we’ve only just scratched the surface of what you can do with Pachyderm. Now that you have a working pipeline, try out some of these commands to explore all of the details associated with the DAG.

For a comprehensive list of operations, check out the Build DAGs section of the documentation or browse the Command Library.

Bonus Exercise

  • How would you update the glob pattern in the video converter pipeline spec (video_mp4_converter.yaml) to only process video files in the raw_videos_and_images repo? That would enable you to reduce the complexity of the user code in def process_video_files and make the pipeline more efficient.


You've successfully completed the Explore Resources, Data, & Logs.