linkedin.com
One of the best return on investment things I ever did, was to sit down and read carefully, back-to-back, a response surface book and a formal experimental analysis book (the opportunity to do so was… | Christos Argyropoulos
One of the best return on investment things I ever did, was to sit down and read carefully, back-to-back, a response surface book and a formal experimental analysis book (the opportunity to do so was provided by the covid lockdowns, not any particular desire of mine, as I avoided the topic during grad school). As I started analyzing bigger and bigger datasets, I used to run in the following (perhaps familiar situation?): a dataset that would fit fine in the memory of a given machine, would run into OOM (Out of Memory issues) as the dataset grew, wasting countless hours of mine and many kwhrs of electricity. I almost never run on this problem now, because I approach the development of performant data analysis pipelines as a designed experiment. One tactic you may consider when you are developing a new dataflow is the three point rule: I typically sample from the 3 dataset, three datasets of increasing size that are separated by powers of 2 (1/2%, 1%, 2% of the dataset) or 10 e.g. 0.1%, 1%, 10%. These 3 points gives one a D-optimal design for the estimation of a quadratic relationship e.g. (Memory) = a0+a1*size+ a2*size^2 (or mutatis mutandis for time). I then run the dataflow a few times (e.g. 2-3) with the lowest dataset size, an equal number with the medium size (another 2-3) and then the same number with the large, randomizing the order of execution and collect performance data during these pilot runs. Fixing the datasets and not taking random samples of the prescribed size will not be a problem here as there will be enough jitter from the operating system scheduler, processor thermal throttling and their interaction with the governor policies) to give one variation to work with. Based on the pilot data I f fit the model and extrapolate to the large dataset I want to analyze. This way I do not waste time with a promising dataflow that will not scale up. The crucial detail is to choose the lowest bound so that the data spill into the main memory from the level 3 cache if interest is in the execution time. If one chooses the smallest dataset to be TOO small (for that you can check the size of the level 3 cache, using something like lscpu | grep -i "L3"), then you may mistake a linear algorithm for a quadratic because your smallest point executed using fast memory, but your largest dataset size relied on DDR). But for OOM detection this does not matter. I typically automate this process by creating and saving these datasets in the disk (if one does so, then one can collect performance data about the disk-to-memory scaling which is a particularly important determinant of performance if you operate out of HDDs (unless you are a YOLO type like me and just use a ridiculous number of HDDs in a RAID0 configuration) and then writing a control script in perl/bash that calls the entire dataflow for each dataset by forking, collects execution times and the RSS (resident set size) and then feeds the data to R for the regression analysis.
