ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


The iPhone SDK: APIs Apple Didn't Want You to Know About

by Jonathan Zdziarski
03/25/2008

Author's Note: Special thanks to Jay Freeman (Saurik) who has done remarkable work in updating the open source tool chain to support iPhone v2.0

With the release of Apple's SDK for building iPhone applications, many have plunged head-first into this new platform for the first time, with the new-found excitement that comes in discovering something entirely new and innovative. The energy surrounding the iPhone has been building steadily since its release last June, and Apple's initial "beta" offering of their SDK gave developers many of the tools they needed to get engaged. Within a short time, however, the community hit a brick wall in many respects, leaving many disenchanted by the restrictions imposed on developers. While Apple insists that the SDK provides the same tools used to create their own software, developers have found that they don't have access to the same low-level functions of the iPhone, such as the ability to run applications in the background, build certain types of objects, or use low-level frameworks such as CoreSurface, Celestial, or LayerKit — all of which provide direct access to graphics and sound components. These, along with many other features, are found in Apple's own applications, but nowhere to be found in the SDK.

A little bit of history is required to fully appreciate this debacle. Since June 2007, the open source community had been hacking on the iPhone and developed a huge audience (estimated to be around 40% of the market). Shortly thereafter, a group of hackers managed to free the iPhone's operating system to run open source software and had succeeded in building their own community SDK (software development kit) for compiling third-party iPhone applications. Since then, the development community surrounding the iPhone has grown considerably, and hundreds of great applications have been made freely available for the device via a popular community software installer. The "Installer" application serves as a kind of "online library," allowing anyone using it to download and install programs directly from the iPhone (over the air), without iTunes or even a desktop machine.

By fall 2007, the rest of the iPhone community had tuned in. Nicholas Penree, author of the popular jailbreakme.com website, had reported that over one million iPhone users had used the site to enable their device for third-party applications. Since then, it has been estimated that over two million iPhone users are now running the third party "Installer" application, giving them full access to the entire public software library for the iPhone. Open source development has become so popular, in fact, that O'Reilly has recently commissioned me to write, and published a book titled "iPhone Open Application Development," which documents many of these APIs and teaches developers how to write applications for the iPhone.

Jump ahead to March 2008. Apple finally realized what a huge financial opportunity they were missing out on when they snubbed third party developers, and decided to release their own version of what the community already had been using for nearly a year, a software development kit (the Apple SDK) and application distribution chain (the iTunes AppStore). Ironically, due to this delay, Apple was surprisingly the one lagging behind the open community, and rather than the open source community duplicating commercial efforts, Apple embarrassingly became the one trying to duplicate the open source community today.

With the introduction of the Apple SDK, developers gauged its functionality based on a comparison to the unofficial, open source SDK released last August. In the process of building this custom, open source compiler for the iPhone, the development community exposed the many low-level APIs (application programming interfaces) available on the device. Using tools such as class-dump, nm, and just plain old trial-and-error gave developers access to the full breadth of functionality available deep within the iPhone's frameworks. It was used to write applications that could look and act just like Apple's preloaded software, so when Apple announced that their SDK was "the same set of tools," many expected that it would look and feel like the open tool chain. Very few had anticipated the many restrictions they've come to find in the official SDK. While roughly 75% of the two SDKs do overlap, the remaining 25% has shown to be very restrictive, removing the developer's ability to do "the real fun stuff" with their application.

iPhone Open Application Development
By Jonathan Zdziarski

 iPhone Open Application Development Book Cover

In this clear and concise book, Jonathan Zdziarski -- one of the original hackers of the iPhone -- explains how developers can design third-party software that will run on this device. You'll learn about iPhone's proprietary development environment, the Objective-C language it uses, and background on the operating system. You also get detailed recipes and working examples for several iPhone features.

Back to the present, the APIs available in the Apple SDK are useful for building your average game, or your average application, but very lacking for building applications with more sophisticated, low-level requirements. Fortunately, there is another set of interfaces that Apple never wanted you to know about, the "real" set of APIs that Apple uses. These are the same interfaces that have been made available through the unofficial SDK "tool chain," only we didn't know it at the time. The great news is that you can use the Apple SDK (via Xcode) to build applications using these hidden APIs, and this article will show you how.

It's important to note that it is unclear whether using these hidden APIs will disqualify your project from being listed in Apple's AppStore. When it comes down to it, the issue is not a technical matter, but rather a licensing and policy issue. Using these APIs can help extend your application's functionality, but be warned that this may also mean you'll need to distribute your application using the community "Installer," or on your own. The good news is that this is what many developers have resolved to do for the sake of writing better software, and with a market penetration of over 40%, the community installer is able to reach a very large audience. It is also believed that the iTunes AppStore will not be available to iPhone users who have unlocked their phones and are running on unauthorized networks, further expanding the potential of the community software installer.

Now for the fun part, for those who are, or will soon be running the Aspen (iPhone OS) software on their iPhone, and would like to build applications using these hidden APIs, there are now two different methods you can use: the open source tool chain or Apple's official SDK (with some customizations). Using either, it is possible to not only write applications that take full advantage of the low-level frameworks used by Apple's own software, but also to build existing applications written for the open source tool chain.

Using Private APIs in the Apple SDK

Apple's low-level APIs, made available in the open tool chain, contain some objects and methods that have been intentionally left out of the Apple SDK, to help control what developers can and can't do. If you think about the ramifications of what it would mean to be able to write your own movie player, for example, it would be very unwise for Apple to foster applications that competed with their own.

Introducing our first example of a missing framework, the CoreSurface framework. CoreSurface allows for direct writes to a screen surface, making applications such as custom movie players and software emulators possible. The framework itself is included with the iPhone and the Apple SDK, as it is used by higher-level libraries such as OpenGL, but developers cannot take advantage of it because the framework headers are missing.

Another example is in a missing set of objects named UIPreferencesTable, used for building settings screens inside an application. Apple's document insists that you create a preference bundle for your application, so that users have to exit your program and use the "Settings" application. This all caters to look-and-feel, but some find it unreasonable, and equate removing this object as removing the ability for a desktop program to have a "Preferences" menu option.

To take advantage of these hidden APIs within the Apple SDK, you'll need to install the header files for these objects and then modify your Xcode projects to use them. An unauthorized set of private headers is being maintained by a group of software developers named "the iPhone Dev Team." Unofficial, these headers are used by the open source SDK. To download and install these low-level APIs, do the following from a Terminal window on your desktop:

$ svn co http://iphone-dev.googlecode.com/svn/branches/include-1.2-sdk

$ cd include-1.2-sdk

$ ./configure
--prefix=/Developer/SDKs/iPhoneOS.sdk/Versions/iPhoneOS2.0.sdk

$ sudo sh install-headers.sh

$ sudo ln -s
/Developer/SDKs/iPhoneOS.sdk/Versions/iPhoneOS2.0.sdk \

 
/Developer/SDKs/iPhoneOS.sdk/Versions/Current

This will install the APIs into a new directory named /Developer/SDKs/iPhoneOS.sdk/Versions/iPhoneOS2.0.sdk. Now you'll need to instruct your Xcode project to use them instead of the "official" APIs. To do this, follow the steps below:

-I/Developer/SDKs/iPhoneOS.sdk/Versions/iPhoneOS2.0.sdk/include

-I/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.0.sdk/usr/include

-I/Developer/Platforms/iPhoneOS.platform/Developer/usr/lib/gcc/arm-apple-darwin9/4.0.1/include

-F/System/library/Frameworks

-F/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.0.sdk/System/library/Frameworks

-F/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.0.sdk/System/library/PrivateFrameworks

-DMAC_OS_X_VERSION_MAX_ALLOWED=1050

If you're using a Makefile instead of Xcode, you can create a very clean build by adding the following to your CFLAGS:

DEV = /Developer/Platforms/iPhoneOS.platform/Developer

SDK = $(DEV)/SDKs/iPhoneOS2.0.sdk

CC = $(DEV)/usr/bin/gcc-4.0

CFLAGS = -arch arm \

-I"/Developer/SDKs/iPhoneOS.sdk/Versions/Current/include" \

-I"$(SDK)/usr/include" \

-I"$(DEV)/usr/lib/gcc/arm-apple-darwin9/4.0.1/include" \

-F"/System/library/Frameworks" \

-F"$(SDK)/System/library/Frameworks"\

-F"$(SDK)/System/library/PrivateFrameworks"

If you build your application, you'll now be using the low-level set of APIs in addition to the standard set. Because the low-level APIs include many private frameworks, such as CoreSurface, you can now import these frameworks into your application.

To use such a framework in Xcode, you'll first need to switch your build to "Device - iPhone OS 2.0". The reason for this is that the simulator does not include many of the frameworks included on the iPhone. In fact, the simulator is unable to even run some "official" SDK applications, such as those using OpenGL. You'll only be able to build and test these low-level frameworks on the iPhone itself.

If you're using a Makefile, you'll want to add these linker settings:

LD = $(CC)

LDFLAGS = -arch arm -lobjc \

-framework CoreFoundation \

-framework Foundation \

-framework UIKit \

-framework CoreSurface \
         
-L"$(SDK)/usr/lib" \
         
-F"$(SDK)/System/library/Frameworks" \

Now that the framework is linked in, add the appropriate includes to your project, for example:

#import <CoreSurface/CoreSurface.h>

Now you're ready to go! Many examples of how to use the many private APIs and frameworks can be found at http://www.iphonedevdocs.com/, as well as in my O'Reilly book. You will write more functional code and sleep better at night knowing that your application isn't restricted by a device manufacturer's policies form-factor requirements!

Building the Open Source Tool Chain

Perhaps you're opposed to the idea of building applications linked in with embedded DRM (digital rights management), or perhaps you just like using open source software. The open source tool chain has been around since August 2007 and has matured considerably in its time. It also happens to run on Linux, Windows, and older versions of Mac OS X, making it a much cheaper alternative for developers who don't want to switch their entire department to Leopard-based Macs.

Many binary distributions of the open source tool chain are floating around. We'll walk you through a simple Leopard installation here that will work on the iPhone OS development platform.

What You'll Need

While there are some unofficial binary distributions of the tool chain floating around the Internet, you'll be building it from sources in this section. The following are requirements for building from source.

Supported Desktop Platform

The first thing you'll need is a desktop platform that is supported. Platforms currently supported by the tool chain are:

Nicholas Penree of Conceited Software took time to adapt the tool chain's installation to run on Leopard, which is what we'll use in our example, with my own notes for adding iPhone OS support. Other platforms follow the same basic steps as these. Official tool chain instructions can be found at http://code.google.com/p/iphone-dev/wiki/Building.

High Speed Internet Connection

The tool chain is several hundred megabytes in size-and that's just the sources. Unless you want to be sitting around for a few days, you'll likely want to download the sources over a high-speed connection. If you don't have one, it might be a good idea to perform the installation from a library or local coffee shop.

Open Source Tools

The next things you'll need are the necessary open source tools installed on your desktop:

If you're missing any of these tools, download and install them before proceeding. On the Mac, these are included with Xcode tools, and you'll want to install or upgrade to the latest version of Xcode before proceeding. Most other operating systems provide them as optional components in their distribution.

Xcode tools can be downloaded from Apple's web site at http://developer.apple.com/tools/xcode/.

Apple SDK

Finally, the last thing you'll need is a copy of the Apple SDK, if you want to use the iPhone OS libraries and frameworks. Your applications will link to these when built.

If you've installed the Apple SDK, just symlink the framework libraries to a place the tool chain will see them:

sudo mkdir -p /usr/local/share

sudo ln -s/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.0.sdk \

/usr/local/share/iphone-filesystem

If you're unable to install these on your machine, you can download these directly from an iPhone running a jailbroken 1.2/2.0 file system; specifically, the libraries and frameworks.

Copying the libraries from your iPhone to your desktop is not condoned by Apple. Check with your local, state, and federal laws to ensure this is legal where you reside.

Presuming you have installed SSH onto the iPhone, use the following commands to download the files you need into a folder called /usr/local/share/iphone-filesystem.

# mkdir -p /usr/local/share/iphone-filesystem

# cd /usr/local/share/iphone-filesystem
 
# mkdir -p ./System/library ./usr

# scp -r root@[IPHONE IP ADDRESS]:/System/library/Frameworks/ .
 
# mv Frameworks ./System/library
 
# scp -r root@iphone:/usr/lib .
 
# mv lib ./usr

Compiling the Tool Chain

The source code for the tool chain is split into two repositories: one for the LLVM compiler framework and one for the rest of the tool chain. Create a build directory for yourself and cd into it. Now use Subversion to check both projects out from their repositories.

$ svn co http://llvm.org/svn/llvm-project/llvm/trunk  llvm-svn -r 42498
 
$ svn co http://iphone-dev.googlecode.com/svn/trunk/ iphone-dev

You'll also need to switch out a couple of SVN repositories to grab the new iPhone OS versions of various pieces. This will eventually be merged into the root of the source tree when iPhone OS is officially released.

$ pushd include
 
$ svn switch http://iphone-dev.googlecode.com/svn/branches/include-1.2-sdk
 
$ popd
 
$ pushd odcctools
 
$ svn switch http://iphone-dev.googlecode.com/svn/branches/odcctools-9.2-ld
 
$ popd

The download may take an hour or longer, depending on the speed of your connection. Once both repositories have been checked out, it's time to start building.

One convention that may be unfamiliar to you is the use of the built-in shell commands pushd and popd. These are similar to the cd command, but they push and pop directories on a stack. This makes it easy to do some work in a new directory and then return to a previous one without having to remember where you were.

You'll also see references to the sudo command. This is a Unix tool providing limited access to run privileged (root) commands. When you want to run a command that requires privileged access (because it accesses sensitive data on the operating system or could have dangerous effects hurting the operating system), type sudo on the command line followed by the command you wish to run. Mac OS X will then prompt you for the root password to your desktop before allowing the command to run. If you don't have sudo, you can safely leave it off of your commands, but will need to first invoke su to become root.

Step 1: Build and install the LLVM framework

The LLVM (Low Level Virtual Machine) framework provides a standard infrastructure for building compilers. It provides the necessary hooks and APIs to build a standardized compiler without having to rewrite all of the basic components of a compiler. Issue the following statements to compile and install a release build of the llvm compiler.

$ pushd llvm-svn
 
$ ./configure --enable-optimized
 
$ make ENABLE_OPTIMIZED=1
 
$ sudo make install
 
$ LLVMOBJDIR=`pwd`
 
$ popd

Step 2: Build and install cross-compiler tools

The following commands build and install the cross-compiler components of the tool chain. These are specific to Mac OS X, so be sure to read the official documentation if you're using a different platform.

$ pushd iphone-dev
 
$ sudo mkdir /usr/local/arm-apple-darwin
 
$ mkdir -p build/odcctools
 
$ pushd build/odcctools
 
$ ../../odcctools/configure --target=arm-apple-darwin --disable-ld64
 
$ export INCPRIVEXT="-isysroot /Developer/SDKs/MacOSX10.4u.sdk"
 
$ make
 
$ sudo make install
 
$ popd
 
$ HEAVENLY=/usr/local/share/iphone-filesystem

Step 3: Install the low-level API headers

Because the iPhone's architecture is different from the desktop's, special headers need to be installed to access the low-level APIs of the iPhone. Issue the following commands to install them:

$ pushd include
 
$ ./configure
--with-macosx-sdk=/Developer/SDKs/MacOSX10.4u.sdk
 
$ sudo bash install-headers.sh
 
$ popd

Step 4: Install the Csu

The Csu provides C hooks into assembly's "start" entry point, and sets up the stack so that your program's main() function can be called. It's essentially glue code.

$ mkdir -p build/csu
 
$ pushd build/csu
 
$ ../../csu/configure --host=arm-apple-darwin
 
$ sudo make install
 
$ popd

Step 5: Build and install llvm-gcc

Now that the LLVM framework, cross-compiler tools, and Csu have been built, the compiler itself can now be built and installed. If you're doing this in stages or have since closed your Terminal window, make sure that you've still got the environment variables $LLVMOBJDIR and $HEAVENLY set to the proper directories. The LLVMOBJDIR variable points to the location of LLVM object files, which were compiled when you built LLVM. These are used to build llvm-gcc. The HEAVENLY variable points to the location where you copied the iPhone's libraries onto your desktop. This directory is used by llvm-gcc to nk to the iPhone's framework and library files when you compile applications. The name "Heavenly" was the code name given by Apple to the 1.0 code base of the iPhone software. The latest version (1.2) is named "Aspen," but the original name is still used in the tool chain. Both are named for ski slopes.

$ set
| grep -e LLVMOBJDIR -e HEAVENLY

If you don't see output from the previous command, you need to set the environment variables again. Get back into your build directory and run:

$ pushd
llvm-svn && LLVMOBJDIR=`pwd` && popd
 
$ HEAVENLY=/usr/local/share/iphone-filesystem

Once you've ensured that these are set, issue the following commands to build and install the compiler.

$ mv llvm-gcc-4.0-iphone/configure llvm-gcc-4.0-iphone/configure.old
 
$ sed \
 
's/^FLAGS_FOR_TARGET=$/FLAGS_FOR_TARGET=${FLAGS_FOR_TARGET-}/g' \
 
 
llvm-gcc-4.0-iphone/configure.old > llvm-gcc-4.0-iphone/configure
 
$ sudo ln -s /usr/local/arm-apple-darwin/lib/crt1.o \
 
 
/usr/local/arm-apple-darwin/lib/crt1.10.5.o
 
$ mkdir -p build/llvm-gcc-4.0-iphone
 
$ pushd build/llvm-gcc-4.0-iphone
 
$ export FLAGS_FOR_TARGET="-mmacosx-version-min=10.1"
 
$ sh ../../llvm-gcc-4.0-iphone/configure \
 
--enable-llvm=`llvm-config --obj-root` \
 
--enable-languages=c,c++,objc,obj-c++ \
 
--target=arm-apple-darwin --enable-sjlj-exceptions \
 
--with-heavenly=$HEAVENLY \
 
--with-as=/usr/local/bin/arm-apple-darwin-as \
 
--with-ld=/usr/local/bin/arm-apple-darwin-ld
 
$ make LLVM_VERSION_INFO=2.0-svn-iphone-dev-0.3-svn
 
$ sudo make install
 
$ popd
 
$ popd

Congratulations! You've built the free tool chain for iPhone. You're now ready to start compiling iPhone applications. The compiler itself can be invoked directly by calling /usr/local/bin/arm-apple-darwin-gcc.

When you build your applications for iPhone OS, you'll need to be sure to use the following additional CFLAGS:

CFLAGS = -fobjc-abi-version=2 \
 
-F/usr/local/share/iphone-filesystem/System/library/PrivateFrameworks

And LDFLAGS:

-lobjc

Resources

Jonathan Zdziarski is better known as the hacker "NerveGas" in the iPhone development community. His work in cracking the iPhone helped lead the effort to port the first open source applications, and his book, iPhone Open Application Development, taught developers how to write applications for the popular device long before Apple introduced its own SDK. Jonathan's website is zdziarski.com.


Return to ONLamp.

Copyright © 2009 O'Reilly Media, Inc.