||Top Ten Data Crunching Tips and Tricks|
|Subject:||"read the input into memory" considered harmful|
I don't agree with point #2, especially the advice to read the input into memory. I work in an environment with large data sets, and I've seen many instances where a script that reads the input into memory is developed on small test data, then unleashed on gigabytes of real data. Either the process exceeds its 4GB address space (most perl executables are still 32-bit), or the server itself runs out of memory.
Showing messages 1 through 1 of 1.
"read the input into memory" considered harmful
2005-06-22 05:49:40 GregWilson [View]