The App Business Blog

Marco wrote a great post explaining the issues with local file storage in iOS 5. Please read it now and then come back.

There are also many threads on the Apple Developer Forums about this (developer login required):

• Securely save files
• Caches folder for content you want to keep?
• Where can I save files on the device that a) are persistent and b) don’t get sync’d with iTunes?
• iCloud iOS Backup taking up a huge amount of space

Confusion

There is considerable confusion around this subject because there are many different issues discussed and lumped together:

• The semantics of cache and tmp directories
• Changes/gaps in Apple’s documentation
• App review rejections
• What is being backed up by iTunes and iCloud
• Changes in behavior in iOS 5

Let’s discuss each issue separately:

The semantics of cache and tmp directories

If you have an OS X or Unix background, it’s easy to understand Apple’s position that such directories have no guarantee as to how long the data in them will persist.

The fact that the tmp directory in the app’s sandbox is not the same as the root /tmp directory, should not make a difference in how you think about this directory.

If you have never observed files being removed from these directories in the past, that is not a guarantee that it will not change in the future. Especially when the change is in accordance with documentation. This is a general rule.

Changes/gaps in Apple’s documentation

As late as June 29, 2011 Apple’s documentation regarding <Application_Home>/Documents said:

Use this directory to store user documents and application data files.

This is pretty clear. No wonder developers are unhappy that the rules for the Documents directory have changed in iOS 5 without any suitable alternative.

And regarding <Application_Home>/Library/Caches

Use this directory to write any application-specific support files that you want to persist between launches of the application or during application updates. Your application is generally responsible for adding and removing these files. It should also be able to re-create these files as needed because iTunes removes them during a full restoration of the device.

The phrasing in this section is definitely vague. The general impression that I get from reading this is that Apple has made a fundamental change with iOS 5 that contradicts this paragraph. But if you instead focus on the words "generally" and "able to re-create" you could argue that Apple has warned you what might happen to your files.

The iOS Data Storage Guidelines state:

To ensure that backups are as efficient as possible, be sure to store your app’s data according to the following guidelines:

1. Only documents and other data that is user-generated, or that cannot otherwise be recreated by your application, should be stored in the <Application_Home>/Documents directory and will be automatically backed up by iCloud.

2. Data that can be downloaded again or regenerated should be stored in the <Application_Home>/Library/Caches directory. Examples of files you should put in the Caches directory include database cache files and downloadable content, such as that used by magazine, newspaper, and map applications.

3. Data that is used only temporarily should be stored in the <Application_Home>/tmp directory. Although these files are not backed up to iCloud, remember to delete those files when you are done with them so that they do not continue to consume space on the user’s device.

If we for the moment disregard the fact that sole purpose of many magazine, newspaper and map apps is to display content offline, these guidelines are clear and make sense.

App review rejections

Developers are reporting that apps that store any/some/much data in the Documents directory are being rejected by App Review.

It’s unlikely that the App Review team has detailed knowledge of which files are being stored in which directory and which of those are user generated vs. data that can be downloaded again or regenerated. Some developers have reported success in responding to the App Review team with an explanation of how their app is storing data and how that is in accordance with the rules.

What is being backed up by iTunes and iCloud

Everything in the app’s home directory is backed up, with the exception of:

• The app bundle itself (.app)
• <Application_Home>/tmp directory
• <Application_Home>/Library/Caches directory

This is clear from the iOS Data Storage Guidelines and Michael Jurewitz comment.

Other documentation clearly states that the Application Support directory is also backed up by iTunes (and presumably iCloud). In the discussions some developers have suggested that Application Support directory would be safer (= more permanent) alternative to Caches. To me it seems that App Review would crack down on large amounts of data stored in Application Support with the same fervor as for the Documents directory, since it’s all about iCloud storage.

Changes in behavior in iOS 5

As of iOS 5, <Application_Home>/Library/Caches may be purged while your app is not running if the device experiences a low disk space warning.

What’s missing?

There is no longer a directory where your app can store files that are:

• Not backed up to iTunes/iCloud
• Not at risk of being purged

It’s obviously too late to implement this for iOS 5.0. But if enough developers make a case that this is really needed for their apps, then it might happen in a future version. Hint: file a bug report.

What’s a developer to do?

If you are currently storing files in the Documents directory

Your app will continue to work in iOS 5 without any issues. Your customers might complain about too much data being backed up to iCloud. (See below.)

However, when you update your app, it’s likely that it will be rejected for storing too much data in the Documents directory.

If you are currently, or are considering, storing files in the Caches directory

Make sure that your app can gracefully handle the situation when any of the files you stored in the Caches directory disappears. One way to handle this is to keep a list of all the files you store in Caches along with their source URLs. (And obviously store this file in a different, more permanent location.) Then at app start go through the list and verify that the files are still there.

If any files are missing you can show a dialog to the customer apologizing, explaining the situation, and asking if the files should be downloaded again. If the device is offline, you apologize and explain that the customer is screwed.

There are many more complicated situations than these two, e.g. where partial data is available. You need to decide what and how much data you can display.

I love the use case pointed out by one of the developers in the Apple forums: His app is used by pilots to display maps. If he was to store the downloaded maps in Caches and they were suddenly deleted by iOS 5 and this was discovered while the plane is in the air, that could have some dire consequences. A dialog asking the pilot to download the maps data again would add insult to injury. Literally.

Migrating existing data

If you update your app to be compliant with the new iOS 5/iCloud rules and now store files in the Caches directory, then you should probably move any existing files from Documents to Caches. I’m sure that the app review team will not test this because they will not have an old version of your app with data saved. But it seems like the right thing to do.

Remember to not start a big job of moving files on the main thread during app startup. This will get your app killed by the startup timer watchdog.

Early warning

When the app is running, you can warn the customer if the device is running low on disk space. This will not prevent files from being removed, but at least it will raise some awareness about the issues.

I don’t know how low the available disk space needs to go before the iOS 5 cleaning process kicks in, and I doubt that Apple will ever specify this. Please add a comment to this post if you have any results from your own experiments.

Let Apple know that this is a big problem

File a bug report.

What’s a customer to do?

Until now, apps that store a lot of data in directories that are backed up have been a problem because the iTunes backup process took a very long time. This was especially true if a large number of files needed to be backed up.

With iCloud backup, customers may not want to use their precious 5 GB data allotment (or pay for additional storage) to backup what they consider to be non-essential data. It is possible to turn off iCloud backup for individual apps. Just go to this easy to find location in the Settings app: iCloud > Storage & Backup > Manage Storage > Backups. In the Backup Options list on this screen, backup can be turned on/off for each app. Since it’s unlikely that your customers will stumble across this setting, you may want to prepare a support email with these instructions.

It is my understanding, although I have not tested this yet, that turning off iCloud backup will not affect iTunes backups. But for customers who have cut the cord and are no longer syncing with their computer, they will be without any backup of their app data if it’s turned off as described above.

written by Nick \\ tags: iOS 5