In reply to "read the input into memory" considered harmful

Article:		Top Ten Data Crunching Tips and Tricks
Subject:		In reply to "read the input into memory" considered harmful
Date:		2005-06-14 08:55:44
From:		jtaylor247

What Greg suggested does not neccessitate a script loading all records into memory first. If you have three functions read(), transform(), and write() you can simply apply these in a read() transform() write() loop that loops once per record leaving only one record in memory at a time.