By using the ulimit command in bash, the limit command in tcsh, or the rlimit command in ksh. See the appropriate manual page for details.
This setting affects all programs run from the shell (directly or indirectly), not the whole system.
If you wish to enable or disable core dumping for all processes by default, you can change the default setting in linux/sched.h. Refer to definition of INIT_TASK, and look also in linux/resource.h.
PAM support optimizes the system's environment, including the amount of memory a user is allowed. In some distributions this parameter is configurable in the /etc/security/limits.conf file. For more information, refer to the Linux Administrator's Security Guide. ("Where Is the Documentation?")