Quick Tip: Update iTunes after you have been running a beta version Pervasent developed app wins “Best Global App” award
Oct 14

Marco wrote a great post explaining the issues with local file storage in iOS 5. Please read it now and then come back.

There are also many threads on the Apple Developer Forums about this (developer login required):

Confusion

There is considerable confusion around this subject because there are many different issues discussed and lumped together:

  • The semantics of cache and tmp directories
  • Changes/gaps in Apple’s documentation
  • App review rejections
  • What is being backed up by iTunes and iCloud
  • Changes in behavior in iOS 5

Let’s discuss each issue separately:

The semantics of cache and tmp directories

If you have an OS X or Unix background, it’s easy to understand Apple’s position that such directories have no guarantee as to how long the data in them will persist.

The fact that the tmp directory in the app’s sandbox is not the same as the root /tmp directory, should not make a difference in how you think about this directory.

If you have never observed files being removed from these directories in the past, that is not a guarantee that it will not change in the future. Especially when the change is in accordance with documentation. This is a general rule.

Changes/gaps in Apple’s documentation

As late as June 29, 2011 Apple’s documentation regarding <Application_Home>/Documents said:

Use this directory to store user documents and application data files.

This is pretty clear. No wonder developers are unhappy that the rules for the Documents directory have changed in iOS 5 without any suitable alternative.

And regarding <Application_Home>/Library/Caches

Use this directory to write any application-specific support files that you want to persist between launches of the application or during application updates. Your application is generally responsible for adding and removing these files. It should also be able to re-create these files as needed because iTunes removes them during a full restoration of the device.

The phrasing in this section is definitely vague. The general impression that I get from reading this is that Apple has made a fundamental change with iOS 5 that contradicts this paragraph. But if you instead focus on the words "generally" and "able to re-create" you could argue that Apple has warned you what might happen to your files.

The iOS Data Storage Guidelines state:

To ensure that backups are as efficient as possible, be sure to store your app’s data according to the following guidelines:

1. Only documents and other data that is user-generated, or that cannot otherwise be recreated by your application, should be stored in the <Application_Home>/Documents directory and will be automatically backed up by iCloud.

2. Data that can be downloaded again or regenerated should be stored in the <Application_Home>/Library/Caches directory. Examples of files you should put in the Caches directory include database cache files and downloadable content, such as that used by magazine, newspaper, and map applications.

3. Data that is used only temporarily should be stored in the <Application_Home>/tmp directory. Although these files are not backed up to iCloud, remember to delete those files when you are done with them so that they do not continue to consume space on the user’s device.

If we for the moment disregard the fact that sole purpose of many magazine, newspaper and map apps is to display content offline, these guidelines are clear and make sense.

App review rejections

Developers are reporting that apps that store any/some/much data in the Documents directory are being rejected by App Review.

It’s unlikely that the App Review team has detailed knowledge of which files are being stored in which directory and which of those are user generated vs. data that can be downloaded again or regenerated. Some developers have reported success in responding to the App Review team with an explanation of how their app is storing data and how that is in accordance with the rules.

What is being backed up by iTunes and iCloud

Everything in the app’s home directory is backed up, with the exception of:

  • The app bundle itself (.app)
  • <Application_Home>/tmp directory
  • <Application_Home>/Library/Caches directory

This is clear from the iOS Data Storage Guidelines and Michael Jurewitz comment.

Other documentation clearly states that the Application Support directory is also backed up by iTunes (and presumably iCloud). In the discussions some developers have suggested that Application Support directory would be safer (= more permanent) alternative to Caches. To me it seems that App Review would crack down on large amounts of data stored in Application Support with the same fervor as for the Documents directory, since it’s all about iCloud storage.

Changes in behavior in iOS 5

As of iOS 5, <Application_Home>/Library/Caches may be purged while your app is not running if the device experiences a low disk space warning.

What’s missing?

There is no longer a directory where your app can store files that are:

  • Not backed up to iTunes/iCloud
  • Not at risk of being purged

It’s obviously too late to implement this for iOS 5.0. But if enough developers make a case that this is really needed for their apps, then it might happen in a future version. Hint: file a bug report.

What’s a developer to do?

If you are currently storing files in the Documents directory

Your app will continue to work in iOS 5 without any issues. Your customers might complain about too much data being backed up to iCloud. (See below.)

However, when you update your app, it’s likely that it will be rejected for storing too much data in the Documents directory.

If you are currently, or are considering, storing files in the Caches directory

Make sure that your app can gracefully handle the situation when any of the files you stored in the Caches directory disappears. One way to handle this is to keep a list of all the files you store in Caches along with their source URLs. (And obviously store this file in a different, more permanent location.) Then at app start go through the list and verify that the files are still there.

If any files are missing you can show a dialog to the customer apologizing, explaining the situation, and asking if the files should be downloaded again. If the device is offline, you apologize and explain that the customer is screwed.

There are many more complicated situations than these two, e.g. where partial data is available. You need to decide what and how much data you can display.

I love the use case pointed out by one of the developers in the Apple forums: His app is used by pilots to display maps. If he was to store the downloaded maps in Caches and they were suddenly deleted by iOS 5 and this was discovered while the plane is in the air, that could have some dire consequences. A dialog asking the pilot to download the maps data again would add insult to injury. Literally.

Migrating existing data

If you update your app to be compliant with the new iOS 5/iCloud rules and now store files in the Caches directory, then you should probably move any existing files from Documents to Caches. I’m sure that the app review team will not test this because they will not have an old version of your app with data saved. But it seems like the right thing to do.

Remember to not start a big job of moving files on the main thread during app startup. This will get your app killed by the startup timer watchdog.

Early warning

When the app is running, you can warn the customer if the device is running low on disk space. This will not prevent files from being removed, but at least it will raise some awareness about the issues.

I don’t know how low the available disk space needs to go before the iOS 5 cleaning process kicks in, and I doubt that Apple will ever specify this. Please add a comment to this post if you have any results from your own experiments.

Let Apple know that this is a big problem

File a bug report.

What’s a customer to do?

Until now, apps that store a lot of data in directories that are backed up have been a problem because the iTunes backup process took a very long time. This was especially true if a large number of files needed to be backed up.

With iCloud backup, customers may not want to use their precious 5 GB data allotment (or pay for additional storage) to backup what they consider to be non-essential data. It is possible to turn off iCloud backup for individual apps. Just go to this easy to find location in the Settings app: iCloud > Storage & Backup > Manage Storage > Backups. In the Backup Options list on this screen, backup can be turned on/off for each app. Since it’s unlikely that your customers will stumble across this setting, you may want to prepare a support email with these instructions.

It is my understanding, although I have not tested this yet, that turning off iCloud backup will not affect iTunes backups. But for customers who have cut the cord and are no longer syncing with their computer, they will be without any backup of their app data if it’s turned off as described above.

written by Nick \\ tags:

13 Responses to “Local file storage in iOS 5”

  1. typeiierror Says:

    Is there any update on this?

  2. FlyingDiver Says:

    iOS 5.0.1 (now available to developers) provides a solution to this issue.

  3. Ashish Sudra Says:

    My Application is also rejected and reason from apple is

    2.23 Apps must follow the iOS Data Storage Guidelines or they will be rejected

    My Application Downloads User Purchased Book in to document directory. And later on use from it.

    Is there any solution ? Please help me .

    Thanks in Advance .

  4. nobre Says:

    As of 5.0.1, there is an extended attribute that may be set on files or folders that indicate “Do not backup to iTunes/iCloud”, so you can keep the files in /Documents or /Library/Application Support, not wasting icloud space or slowing down itunes backup, but not risking being purged by the system. Keep in mind that /Library/Caches can still be purged, regardless if this flag is set or not.

  5. MikeR Says:

    Yep, there is an update on this…see the following page at Apple…https://developer.apple.com/library/ios/#qa/qa1719/_index.html …you will need to login with your developers Apple ID, but in summary, Apple has introduced a new “do not backup” attribute that can be applied to any file or folder. If this attribute is set, the file will be persisted even in low disk storage conditions, and the file(s)/folder(s) will not be backed up…It is worth noting that this solution requires iOS 5.0.1 at a minimum.

  6. tstar Says:

    This is very disrespectful act from Apple. They want to control everything and we developers will follow their stupid guidelines and keep using iOS to make money. This will further make Apple take more control by defining their own rules.
    Anyways I hope this issue gets fixed in next update.

  7. Chris Says:

    There is an update on the post at Marco’s site – the first link quoted in the article. Apparently as of 5.0.1, you can mark “large/redownloadable/critical files” in Documents as ‘do not backup’, which would appear to go some way to resolving the problems discussed.

  8. SVK Says:

    As Marco mentions in an update to his post (at the bottom), this was fixed in 5.0.1, see:

    https://developer.apple.com/library/ios/#qa/qa1719/_index.html

  9. jerry Says:

    There is a solution to this as of iOS 5.0.1 where Apple allows you to set a flag on files and directories to indicate that they should NOT be backed up. You can still store in the Documents directory and not worry about app rejection. The tech note is here: http://developer.apple.com/library/ios/#qa/qa1719/_index.html#//apple_ref/doc/uid/DTS40011342

  10. song soknay Says:

    how can I create sub folder in rootDirectory with device ipad?

  11. Lee Probert Says:

    I’ve just been rejected for the 2nd time for storing a DB file in /Documents and then after seeing that the Google Analytics iOS SDK had moved their location to /Library I tried moving my DB there too … rejected again.

    I will now need to move my DB file to /Caches but I am now concerned that the existence of the Google analytics file in /Library is going to be an issue.

    Doh!

  12. David Price Says:

    Just found out about this for our apps.
    Seen it (very) roughly happen at about under 2GB free on a 16GB iPad2 device

  13. Magdahar Says:

    add NSURLIsExcludedFromBackupKey to files you store in the documents directory and they won’t be stored on the cloud

    - (BOOL) addSkipBackupAttributeToItemAtPath:(NSString *)aPath {

    NSURL *URL = [NSURL fileURLWithPath:aPath] ;
    if (![[NSFileManager defaultManager] fileExistsAtPath: [URL path]]) {
    return YES ;
    }

    NSError *error = nil;
    BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES] forKey: NSURLIsExcludedFromBackupKey error: &error];
    if(!success){
    // NSLog(@”Error excluding %@ from backup %@”, [URL lastPathComponent], error);
    }
    return success ;
    }

Leave a Reply