Document Type
Article
Date of Original Version
2019
Department
Computer Science and Statistics
Abstract
Both astronomy and biology are experiencing explosive growth of data, resulting in a “big data” problem that stands in the way of a “big data” opportunity for discovery. One common question asked of such data is that of approximate search (ρ–nearest neighbors search). We present a hierarchical search algorithm for such data sets that takes advantage of particular geometric properties apparent in both astronomical and biological data sets, namely the metric entropy and fractal dimensionality of the data. We present CHESS (Clustered Hierarchical Entropy-Scaling Search), a search tool with virtually no loss in specificity or sensitivity, demonstrating a 13.6 × speedup over linear search on the Sloan Digital Sky Survey’s APOGEE data set and a 68 × speedup on the GreenGenes 16S metagenomic data set, as well as asymptotically fewer distance comparisons on APOGEE when compared to the FALCONN locality-sensitive hashing library. CHESS demonstrates an asymptotic complexity not directly dependent on data set size, and is in practice at least an order of magnitude faster than linear search by performing fewer distance comparisons. Unlike locality-sensitive hashing approaches, CHESS can work with any user-defined distance function. CHESS also allows for implicit data compression, which we demonstrate on the APOGEE data set. We also discuss an extension allowing for efficient k-nearest neighbors search.
Citation/Publisher Attribution
N. Ishaq, G. Student and N. M. Daniels, "Clustered Hierarchical Entropy-Scaling Search of Astronomical and Biological Data," 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, 2019, pp. 780-789, doi: 10.1109/BigData47090.2019.9005688.
Available at: http://dx.doi.org/10.1109/BigData47090.2019.9005688
Author Manuscript
This is a pre-publication author manuscript of the final, published article.
Terms of Use
This article is made available under the terms and conditions applicable
towards Open Access Policy Articles, as set forth in our Terms of Use.