Understanding Network I/O, Part 2
Subject:   Mixing C and Python.
Date:   2004-02-13 11:20:25
From:   jbi130
Before I discovered Python I wrote an event driven server in C (using libevent). At first it was mainly I/O bound so it worked out great. Then came along some CPU intensive tasks, so I spin off a few threads. They are pretty localized so my only locks are for passing events in and out of the threads.

Now I'm looking at doing more work with Python. Whats the ideal mix of Python and C. I'm looking at using Python modules to extend my C app.

Obviously the other way is to use Python and extend in C. If my C exensions are working in threads how will this affect Python and the GIL?

I'd probably keep all I/O event based and in the main thread and just use worker threads to do the long running tasks.

So now I have to decide, C as the main loop or Python/Twisted.

Full Threads Oldest First

Showing messages 1 through 2 of 2.

  • Mixing C and Python.
    2004-02-23 00:15:24  belotsky [View]

    Do not give up on threads altogether. As the research quoted in the article shows, threads are still a very valuable technique in many cases.

    Using Python from multiple threads, however, can impose significant overhead due to contention for the GIL. For example, if you embed Python, your threads cannot safely access the interpreter without holding the GIL. See Thread State and the Global Interpreter Lock in the Python/C API Reference Manual.

    An alternative is to use processes instead of threads in your application. Such an approach is particularly promising for computationally intensive tasks on multi-processor architectures. There was an interesting talk on this topic at PyCon last year. The ideas and results presented by the authors (see previous link) should be helpful to you. You may also be able to use their project (called POSH) directly, although it is still in its early stages.

    In any case, it is important to take the time to think about the implications of the various available techniques in your specific situation. It is also highly beneficial -- and usually essential -- to experiment with different approaches before making a major design decision. Above all, do not feel pressured to throw away your work just because it is not entirely conventional, or requires a less popular methodology for successful completion.
  • Mixing C and Python.
    2004-02-14 01:00:29  spiv [View]

    I'd use Twisted for the main loop, definitely. You can drop the GIL in your C code using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros -- see the Python C API docs for details.

    See also for a rant about why you should extend, not embed :)