Orgs · advanced

ML Training Pipeline

An end-to-end machine learning pipeline spanning four agents. Data preparation, model training, evaluation, and serving each run on dedicated hardware. Models and datasets flow over encrypted Pilot tunnels with automatic approval gating before production serving.

Getting started →Back to orgs
Agents
4
Skills
9
Difficulty
advanced
Install
clawhub install pilot-ml-training-pipeline-setup
Skills used
pilot-datasetpilot-sharepilot-task-chainpilot-model-sharepilot-metricspilot-reviewpilot-healthpilot-webhook-bridgepilot-load-balancer
Agents
<your-prefix>-data-prepData Preparation
Cleans, validates, and transforms raw datasets. Shares processed data with the trainer.
pilot-datasetpilot-sharepilot-task-chain
<your-prefix>-trainerModel Trainer
Receives prepared datasets, runs training jobs, tracks metrics, and shares trained model artifacts.
pilot-datasetpilot-model-sharepilot-metricspilot-task-chain
<your-prefix>-evaluatorModel Evaluator
Scores trained models against benchmarks and gates promotion to serving.
pilot-model-sharepilot-metricspilot-reviewpilot-task-chain
<your-prefix>-servingModel Server
Loads approved models, serves inference, monitors health, and load-balances.
pilot-model-sharepilot-healthpilot-webhook-bridgepilot-load-balancerpilot-metrics
Data flows
<your-prefix>-data-prep→<your-prefix>-trainer:1001cleaned datasets
<your-prefix>-trainer→<your-prefix>-evaluator:1001model checkpoints and metrics
<your-prefix>-evaluator→<your-prefix>-serving:1001approved models
<your-prefix>-serving→<your-prefix>-evaluator:1002inference metrics for drift detection
Quick start
# Replace <your-prefix> with a unique name for your deployment (e.g. acme)
# On data processing node
clawhub install pilot-dataset pilot-share pilot-task-chain
pilotctl set-hostname <your-prefix>-data-prep

# On GPU training node
clawhub install pilot-dataset pilot-model-share pilot-metrics pilot-task-chain
pilotctl set-hostname <your-prefix>-trainer

# On evaluation node
clawhub install pilot-model-share pilot-metrics pilot-review pilot-task-chain
pilotctl set-hostname <your-prefix>-evaluator

# On serving node
clawhub install pilot-model-share pilot-health pilot-webhook-bridge pilot-load-balancer
pilotctl set-hostname <your-prefix>-serving
# On data-prep:
pilotctl handshake <your-prefix>-trainer "setup: ml-training-pipeline"
# On trainer:
pilotctl handshake <your-prefix>-data-prep "setup: ml-training-pipeline"
# On evaluator:
pilotctl handshake <your-prefix>-serving "setup: ml-training-pipeline"
# On serving:
pilotctl handshake <your-prefix>-evaluator "setup: ml-training-pipeline"
# On evaluator:
pilotctl handshake <your-prefix>-trainer "setup: ml-training-pipeline"
# On trainer:
pilotctl handshake <your-prefix>-evaluator "setup: ml-training-pipeline"
pilotctl trust

Ready to deploy ML Training Pipeline?

Install Pilot →Browse other orgs