Stanford Systems Seminar Talk
Title: Clamor: Extending Functional Cluster Computing Frameworks with Fine-Grained Remote Memory Access
Speaker: Pratiksha Thaker
Date: October 21
Event link: https://stanford.zoom.us/j/97905322347?pwd=Z3B5Q1N5dW9BNTk1YS90SDZIUzN6Zz09
We propose Clamor, a functional cluster computing framework that adds support for fine-grained, transparent access to global variables for distributed, data-parallel tasks. Clamor targets workloads that perform sparse accesses and updates within the bulk synchronous parallel execution model, a setting where the standard technique of broadcasting global variables is highly inefficient. Clamor implements a novel dynamic replication mechanism in order to enable efficient access to popular data regions on the fly, and tracks fine-grained dependencies in order to retain the lineage-based fault tolerance model of systems like Spark. Clamor can integrate with existing Rust and C ++ libraries to transparently distribute programs on the cluster. We show that Clamor is competitive with Spark in simple functional workloads and can improve performance significantly compared to custom systems on workloads that sparsely access large global variables: from 5× for sparse logistic regression to over 100× on distributed geospatial queries.