Article:
  Keeping Your Life in Subversion
Subject:   Subversion cannot handle ~100 MB files
Date:   2005-01-07 14:39:55
From:   sanchonevesgraca
You expect to commit more and more of your home directory to Subversion repositories. Currently, Subversion cannot handle files with size of the order of 100 MB in a realistic time interval (files of this size are seen in image processing projects). Checking out a few large files locally takes about 10 minutes but over HTTP from the Subversion server to a client machine takes hours.
Full Threads Oldest First

Showing messages 1 through 3 of 3.

  • Subversion cannot handle ~100 MB files
    2005-01-07 15:52:28  joeyh [View]

    As a computer programmer, I don't generally edit large files. While I do have several gigabytes of data in svn, it's in hundreds of thousands of small files, with a large file being on the order of 5 or 10 mb. With this workload I find svn to be reasonably fast, even if I'm doing an update of my entire home directory. These is some overhead compared to rsync, but I can live with that.

    Anyway, I did some benchmarking here with 200 mb files containing random data, and it takes me 11 minutes to check such a file in locally to a repository checked out with file://, 10 minutes to check such a file out over svn+ssh:// to a remote machine over 802.11b wireless, 9 minutes to scp the same file over the same link, and 7 minutes to check it out locally. These don't seem to match your numbers, but then you're using the http:// protocol which is probably much less efficient and anyway the network operations appeared bandwidth/disk-bound to me.

    Hope this helps, but you can probably get better information on subversion scalability with larger files on one of the subversion mailing lists.
    • Subversion cannot handle ~100 MB files
      2005-01-08 06:59:16  sanchonevesgraca [View]

      Large files are an issue also for programmers: consider the scenario of checking out large project dependencies such as libraries. The HTTP protocol is widely used by Subversion projects, since it allows connection to repositories through firewalls. So the limitation I pointed out is not a minor problem. The long checkout times I observed occurred over HTTP in internet and intranet connectivity so the limitation was not because of the network bandwidth nor can it be explained by the HTTP protocol. I also noticed that the svn client would, after about ten minutes, take all the computer processor capability and the transfer speed would be quite small (~5 KBps). I do not know the internals of the Subversion checkout implementation, but these observations suggest that the design does not scale up to this file size. Others have encountered this limitation (see http://svn.haxx.se/dev/archive-2004-07/0801.shtml).
      • Subversion cannot handle ~100 MB files
        2005-11-25 03:38:04  aditibhate [View]

        Can any body please tell me ,do you face same problem when subversion is configured on TCP/IP.
        If so then how much time does it take?
        and if it doesn't take so much time ..how far is it difficult to configure subversion on TCP/IP.

        anticipating an earliest reply
        this is a bit urgent
        Aditi