Menu

Writing HTCondor DAG Files: Job Dependencies, Variables, and Concurrency Limits
📰

Writing HTCondor DAG Files: Job Dependencies, Variables, and Concurrency Limits

DEV Community: tutorial·Haruki Niwa·3 days ago
#k9SXqSEJ
#devtotutorial#devto#dev#submit#exit
Reading 0:00
15s threshold

Part 4 of 6 in Building and Operating an HTCondor Pool for Research Computing.

Our lab’s GPU tutorial defines three PyTorch runs with different training parameters. They share one submit file, and DAGMan admits one of the three nodes to the queue at a time. In a participant analysis, the final job also needs every participant’s result, even when those results come from different workers.

This walkthrough starts with two Linux jobs and one line of data you can follow from worker to worker. It then applies the same file-transfer rules to the MAT-files from Part 3. The commands target HTCondor 25.0 LTS on the Ubuntu access point from Part 1. The outputs described below are what to check in your own run.

1. Pass a result from A to B

A DAG is a directed graph with no cycles. Each JOB line names a node and gives its HTCondor submit file. A PARENT … CHILD … line makes the child eligible for submission after all its parents succeed.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.