Beware UserDefaults: a tale of hard to find bugs, and lost data

October 5, 2024

An iPhone sitting on an open book showing its Lock Screen.

Excuse the alarmist title, but I think it’s justified, as it’s an issue that’s caused me a ton of pain in both support emails and actually tracking it down, so I want to make others aware of it so they don’t similarly burned.

Brief intro

For the uninitiated, UserDefaults (née NSUserDefaults) is the de facto iOS standard for persisting non-sensitive, non-massive data to “disk” (AKA offline). In other words, are you storing some user preferences, maybe your user’s favorite ice cream flavors? UserDefaults is great, and used extensively from virtually every iOS app to Apple sample code. Large amount of data, or sensitive data? Look elsewhere! This is as opposed to just storing it in memory where if the user restarts the app all the data is wiped out.

It’s a really handy tool with a ton of nice, built-in things for you:

  • No needing to mess with writing to files yourself, and better yet, no need to coordinate when to persist values back to the disk
  • Easy to share data between your app’s main target and secondary targets (like a widget target)
  • Automatic serialization and deserialization: just feed in a String, Date, Int, and UserDefaults handles turning it into bytes and back from bytes
  • Thread-safe!

So it’s no wonder it’s used extensively. But yeah, keep the two limitations in mind that Apple hammers home:

Okay, so what’s the problem

Turns out, sometimes you can request your saved data back from UserDefaults and it… just won’t have it! That’s a pretty big issue for a system that’s supposed to reliably store data for you.

This can amount to an even bigger issue that leads to permanent data loss.

Imagine a situation where a user has been meticulously opening your app for 364 days in a row. On day 365, your app promised a cool reward! When the user last closed the app, you stored 364 to UserDefaults.

The user wakes up on day 365, excited for their reward:

  1. App launches
  2. App queries UserDefaults for how many days in a row the user has opened the app
  3. App returns 0 (UserDefaults is mysteriously unavailable so its API returns the default integer value of 0)
  4. It’s a new day, so you increment that value by 1, so that 0 changes to 1
  5. Save that new value back to UserDefaults

Now, instead of your user having a fun celebration, their data has been permanently overwritten and reset! They are having a Sad Day™.

It basically means, if at any point you trust UserDefaults to accurately return your data (which you know, sounds like a fair assumption) you might just get incorrect data, which you then might make worse by overwriting good data with.

And remember, you’re not meant to store sensitive data in UserDefaults, but even if it’s not sensitive data it might be valuable. The user’s day streak above is not sensitive data that would be bad if leaked online like a password, but it is valuable to that user. In fact I’d argue any data persisted to the disk is valuable, otherwise you wouldn’t be saving it. And you should be always be able to trust an API to reliably save your data.

What??? How is this happening? 😵‍💫

As I understand it, there’s basically two systems coming together (and working incorrectly, if you ask me) to cause this:

1. Sensitive data encryption

When using Keychain or files directly, as a developer you can mark data that should be encrypted until the device is unlocked by Face ID/Touch ID/passcode. This way if you’re storing a sensitive data like a token or password on the device, the contents are encrypted and thus unreadable until the device is unlocked.

This meant if the device was still locked, and you, say, had a Lock Screen Widget that performed an API request, you would have to show placeholder data until the user unlocked the device, because the sensitive data, namely the user’s API token, was encrypted and unable to be used by the app to fetch and show data until the user unlocked the device. Not the end of the world, but something to keep in mind for secure data like API tokens, passwords, secrets, etc.

2. Application prewarming

Starting with iOS 15, iOS will sometimes wake up your application early so that when a user launches it down the road it launches even quicker for them, as iOS was able to do some of the heavy lifting early. This is called prewarming. Thankfully per Apple, your application doesn’t fully launch, it’s just some processes required to get your app working:

Prewarming executes an app’s launch sequence up until, but not including, when main() calls UIApplicationMain(::::).

Okay, so what happened with these two?

It seems at some point, even though UserDefaults is intended for non-sensitive information, it started getting marked as data that needs to be encrypted and cannot be accessed until the user unlocked their device. I don’t know if it’s because Apple found developers were storing sensitive data in there even when they shouldn’t be, but the result is even if you just store something innocuous like what color scheme the user has set for your app, that theme cannot be accessed until the device is unlocked.

Again, who cares? Users have to unlock the device before launching my app, right? I thought so too! It turns out, even though Apple’s prewarming documentation states otherwise, developers have been reporting for years that that’s just wrong, and your app can effectively be fully launched at any time, including before the device is even unlocked.

Combining this with the previous UserDefaults change, you’re left with the above situation where the app is launched with crucial data just completely unavailable because the device is still locked.

UserDefaults also doesn’t make this clear at all, which it could do by for instance returning nil when trying to access UserDefaults.standard if it’s unavailable. Instead, it just looks like everything is as it should be, except none of your saved keys are available anymore, which can make your app think it’s in a “first launch after install” situation.

The whole point of UserDefaults is that it’s supposed to reliably store simple, non-sensitive data so it can be accessed whenever. The fact that this has now changed drastically, and at the same time your app can be launched effectively whenever, makes for an incredibly confusing, dangerous, and hard to debug situation.

And it’s getting worse with Live Activities

If you use Live Activities at all, the cool new API that puts activities in your Dynamic Island and Lock Screen, it seems if your app has an active Live Activity and the user reboots their device, virtually 100% of the time the above situation will occur where your app is launched in the background without UserDefaults being available to it. That means the next time your user actually launches the app, if at any point during your app launching you trusted the contents of UserDefaults, your app is likely in an incorrect state with incorrect data.

This bit me badly, and I’ve had users email me over time that they’ve experienced data loss, and it’s been incredibly tricky to pinpoint why. It turns out it’s simply because the app started up, assuming UserDefaults would return good data, and when it transparently didn’t, it would ultimately overwrite their good data with the returned bad data.

I’ve talked to a few other developers about this, and they’ve also reported random instances of users being logged out or losing data, and after further experimenting been able to now pinpoint that this is what caused their bug. It happened in past apps to me as well (namely users getting signed out of Apollo due to a key being missing), and I could never figure out why, but this was assuredly it.

If you’ve ever scratched your head at a support email over a user’s app being randomly reset, hopefully this helps!

I don’t like this ☹️

I can’t overstate what a misstep I think this was. Security is always a balance with convenience. Face ID and Touch ID strike this perfectly; they’re both ostensibly less secure per Apple’s own admission than, say, a 20 digit long password, but users are much more likely to adopt biometric security so it’s a massive overall win.

Changing UserDefaults in this way feels more on the side of “Your company’s sysadmin requiring you to change your password every week”: dubious security gains at the cost of user productivity and headaches.

Further, it’s not as if UserDefaults is truly secure with this change. It’s only encrypted between the time the device is rebooted and first unlocked. So sure, if you shut down your phone and someone powers it back up, the name of your pet cow might be knowable to the attacker, but literally any time after you unlock the device after first boot, even if you re-lock it, Mr. Moo will be unencrypted.

Update: After more investigating it seems like this change has been around for years and years, but the results of it have just has gotten more prevalent with things like prewarming and Live Activities.

I mean no shade to the nice folks at Apple working on UserDefaults, I’m sure there’s a lot of legacy considerations to take into account that I cannot fathom that makes this a complex machine. I’m just sad that this lovely, simple API has some really sharp edges in 2024.

But enough moaning, let’s fix it.

Solution 1

Because iOS is now seemingly encrypting UserDefaults, the easiest solution is to check UIApplication.isProtectedDataAvailable and if it returns false, subscribe to NotificationCenter for when protectedDataDidBecomeAvailableNotification is fired. This was previously really useful for knowing when Keychain or locked files were accessible once the device was unlocked, but it now seemingly applies to UserDefaults (despite not being mentioned anywhere in its documentation or UserDefault’s documentation 🙃).

I don’t love this solution, because it effectively makes UserDefaults either an asynchronous API (“Is it available? No? Okay I’ll wait here until it is.”), or one where you can only trust its values sometimes, because unlike the Keychain API for instance, UserDefaults API itself does not expose any information about this when you try to access it when it’s in a locked state.

Further, some developers have reported UserDefaults still being unavailable even once isProtectedDataAvailable returns true, presumably where (for one reason or another) UserDefaults isn’t reading the disk store back into memory even though the file became available.

Further further, UIApplication.isProtectedDataAvailable is just not a great API for this use case. For one, it’s weirdly only available in a main app context, so if you’re executing code from a widget extension and want to check if protected data is available, you’ll have to resort to some other method, like trying to write to disk and read it back. And on top of this, isProtectedDataAvailable returns false when the device is locked, which will cause you to think UserDefaults is also unavailable. But remember, UserDefaults is only unavailable before the first lock upon reboot, any time after that it’s totally accessible, and this API will still return false in those cases as UserDefaults doesn’t follow the strict protection mechanism that that API tracks.

Solution 2

For the mentioned reasons, I don’t really like/trust Solution 1. I want a version of UserDefaults that acts like what it says on the tin: simply, quickly, and reliably retrieve persisted, non-sensitive values. This is easy enough to whip up ourselves, we just want to keep in mind some of the things UserDefaults handles nicely for us, namely thread-safety, shared between targets, and an easy API where it serializes data without us having to worry about writing to disk. Let’s quickly show how we might approach some of this.

UserDefaults is fundamentally just a plist file stored on disk that is read into memory, so let’s create our own file, and instead of marking it as requiring encryption like iOS weirdly does, we’ll say that’s not required:

// Example thing to save
let favoriteIceCream = "chocolate"

// Save to your app's shared container directory so it can be accessed by other targets outside main
let appGroupID = ""

// Get the URL for the shared container
guard let containerURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: appGroupID) else {
    fatalError("App Groups not set up correctly")
}

// Create the file URL within the shared container
let fileURL = containerURL.appendingPathComponent("Defaults")
    
do {
    let data = favoriteIceCream.data(using: .utf8)
    try data.write(to: fileURL)

    // No encryption please I'm just storing the name of my digital cow Mister Moo
    try FileManager.default.setAttributes([.protectionKey: .none], ofItemAtPath: fileURL.path)
    print("File saved successfully at \(fileURL)")
} catch {
    print("Error saving file: \(error.localizedDescription)")
}

(Note that you could theoretically modify the system UserDefaults file in the same way, but Apple documentation recommends against touching the UserDefaults file directly.)

Next let’s make it thread safe by using a DispatchQueue.

private static let dispatchQueue = DispatchQueue(label: "DefaultsQueue")

func retrieveFavoriteIceCream() -> String? {
   return dispatchQueue.sync { 
      guard let containerURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: "app-group-id") else { return nil }

      let fileURL = containerURL.appendingPathComponent(fileName)
            
      do {
         let data = try Data(contentsOf: fileURL)
         return String(data: data, encoding: .utf8)
      } catch {
         print("Error retrieving file: \(error.localizedDescription)")
         return nil
      }
   }
}

func save(favoriteIceCream: String) {
   dispatchQueue.sync { 
      guard let containerURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: "app-group-id") else { return }

      let fileURL = containerURL.appendingPathComponent(fileName)

      do {
         let data = favoriteIceCream.data(using: .utf8)
         try data.write(to: fileURL)
         try FileManager.default.setAttributes([.protectionKey: .none], ofItemAtPath: fileURL.path)
         print("File saved successfully at \(fileURL)")
      } catch {
         print("Error saving file: \(error.localizedDescription)")
      }
   }
}

(You probably don’t need a concurrent queue for this, so I didn’t.)

But with that we have to worry about data types, let’s just make it so long as the type conforms to Codable we can save or retrieve it:

func saveCodable(_ codable: Codable, forKey key: String) {
    do {
        let data = try JSONEncoder().encode(codable)
        // Persist raw data bytes to a file like above
    } catch {
        print("Unable to encode \(codable): \(error)")
    }
}

func codable<T: Codable>(forKey key: String, as type: T.Type) -> T? {
    let data = // Fetch raw data from disk as done above
    
    do {
        return try JSONDecoder().decode(T.self, from: data)
    } catch {
        print("Error decoding \(T.self) for key \(key) with error: \(error)")
        return nil
    }
}

// Example usage:
let newFavoriteIceCream = "strawberry"
saveCodable(newFavoriteIceCream, forKey: "favorite-ice-cream")

let savedFavoriteIceCream = codable(forKey: "favorite-ice-cream", as: String.self)

Put those together, wrap it in a nice little library, and bam, you’ve got a UserDefaults replacement that acts as you would expect. In fact if you like the encryption option you can add it back pretty easily (don’t change the file protection attributes) and you could make it clear in the API when the data is inaccessible due to the device being locked, either by throwing an error, making your singleton nil, awaiting until the device is unlocked, etc.

Solution 3

Update! I wrote a little library to kinda wrap all of the above into a handy little tool! It’s called TinyStorage.

End

Maybe this is super obvious to you, but I’ve talked to enough developers where it wasn’t, that I hope in writing this it can save you the many, many hours I spent trying to figure out why once in a blue moon a user would be logged out, or their app state would look like it reset, or worst of all: they lost data.