HPC Data Pipeline.
A scalable cloud-based data processing and computing platform for large-volume scientific data — designed to take cryo-electron microscopy from instrument to 3D model in a fraction of the time of conventional sync workflows.
Abstract
A multi-stage data transfer pipeline that introduces a dynamically generated filter into the prepare phase of cloud sync, limiting scans to only the files present at the staging location rather than recursing the full source and destination. Combined with on-demand FSx Lustre provisioning, multi-queue GPU and CPU scheduling, and a self-service dataset management utility, the system collapses the time from microscope to interpretable 3D structure.