Speaker
Description
With the evolution of the WLCG towards opportunistic resource usage and cross-site data access, new challenges for data analysis have emerged in recent years. To enable performant data access without relying on static data locality, distributed caching aims at providing data locality dynamically. Recent work successfully employs various approaches for effective and coherent caching, from centrally managed approaches employing volatile storage to hybrid approaches orchestrating autonomous caches. However, due to the scale and use-case of the WLCG there is little prior work to assess the general applicability and scalability of these approaches.
Building on our previous developments of coordinated, distributed caches at KIT, we have identified the primary challenge not in the technical implementation but the underlying logic and architecture. We have studied several key issues solved by all approaches in various ways: aggregation of meta-data, identification of viable data, coherence of cache contents, and integration of temporary caches. Monitoring data from XRootD storage and HTCondor batch systems from both our Tier 1 and Tier 3 infrastructure provides realistic usage characteristics. This allows us to assess both the use cases of well-defined user jobs, as well as late-bound anonymous pilot jobs.
In this contribution, we present our findings on the implications of different architectures for distributed caching depending on the targeted use-case.
| Consider for promotion | No |
|---|
