Beer Diary version 1.8 released

Version 1.8 of the BeerDiary app is now live on the iOS AppStore! You can download it here.

I would love to receive your feedback. If you have questions or remarks don’t hesitate to send me a message.

New features

iCloud sync

This version adds iCloud sync for all your data. It is now possible to use the app on multiple devices while your data is automatically kept in sync.

When a sync is in progress, a small progress indicator will be shown. In case things are broken, it will turn into a failure icon and you can click on it to get some hints on how the situation should be resolved. Hopefully you will not need that 😁.

This feature has been high on my wish list for a while and it took quite a bit of time and effort to get it just right. I hope you enjoy it!

‘Select All’ in Edit mode

Because the focus of this version was iCloud, only one small usability improvement was added. There is now a ‘Select All’ button in Edit mode that allows you to select and deselect all beers at once.

Implementation notes

Feel free to skip this section if you’re not that technical 😁

iCloud Sync

It took more work than expected to integrate this feature. One of the issues I had was that information on this topic is pretty fragmented.

These resources were very helpful:

1. Apple WWDC video: Using Core Data With CloudKit - WWDC19 - Videos - Apple Developer
2. Apple WWDC video: Sync a Core Data store with the CloudKit public database - WWDC20 - Videos - Apple Developer
3. Fatbobman’s series of excellent blog posts
4. A nice basic tutorial: Sharing Core Data With CloudKit in SwiftUI - Kodeco
5. If you want to go more in depth, Core Data by Tutorials is a really nice book.
6. Modern, Efficient Core Data - Kodeco
7. Even though the name might be misleading, this sample project also includes a CoreData version indicating best practices: Adopting SwiftData for a Core Data app
8. TN3163: Understanding the synchronization of NSPersistentCloudKitContainer
9. TN3164: Debugging the synchronization of NSPersistentCloudKitContainer
10. TN3162: Understanding CloudKit throttles
11. CloudKit With CoreData Not Working in Production
12. iCloud Sync Doesn’t Work in Production: CoreData and CloudKit Issues

In the end I had to combine the above data and information from StackOverflow posts and any books I could find to get to a working feature. This could have been so much easier IMHO.

Configuring NSPersistentCloudKitContainer

To get sync up and running, you need to configure NSPersistentCloudKitContainer just right or else things might A) not work or B) break in subtle ways.

Step 1 - Set up NSPersistentStoreDescription

Here is an extension to get the App Group store URL

public extension URL {

    /// Returns a URL for the given app group and database pointing to the sqlite database.
    static func storeURL(for appGroup: String, databaseName: String) -> URL {
        guard let container = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: appGroup) else {
            fatalError("Shared file container could not be created.")
        }

        return container.appendingPathComponent("\(databaseName).sqlite")
    }
}

And here is how to set up the container:

// Configure store URL
let storeURL = URL.storeURL(for: "<app group identifier>",
                            databaseName: "<database name>")

// Construct NSPersistentStoreDescription
let storeDescription = NSPersistentStoreDescription(url: storeURL)

// Enable Persistent History Tracking
// It helps to track changes, sync with CloudKit and help resolve conflicts
storeDescription.setOption(true as NSNumber,
                            forKey: NSPersistentHistoryTrackingKey)

// Enable remote change notifications
storeDescription.setOption(true as NSNumber,
                            forKey: NSPersistentStoreRemoteChangeNotificationPostOptionKey)

// Enable CloudKit sync unless in unit test scenario
if syncEnabled {
    // Set up CloudKit container identifier
    let options = NSPersistentCloudKitContainerOptions(containerIdentifier: containerIdentifier)
    options.databaseScope = .private
    storeDescription.cloudKitContainerOptions = options
}

container.persistentStoreDescriptions = [storeDescription]

⚠️ Note that sync is disabled if running a unit test. Even though my tests all use in-memory stores, the test run using the actual app as a host. There would be no point in syncing the actual data store at this time.

Step 2 - Configure view context

This should be done after you have called container.loadPersistentStores.

// This will be useful later on when for instance widgets will also access the store
container.viewContext.transactionAuthor = "app"

// Context automatically merges changes saved to its persistent store coordinator or parent context.
container.viewContext.automaticallyMergesChangesFromParent = true

// In-memory changes have priority over other versions
container.viewContext.mergePolicy = NSMergePolicy.mergeByPropertyObjectTrump

#if DEBUG
// Run initializeCloudKitSchema() once to update the CloudKit schema every time you change the Core Data model.
// Dispatch to the next runloop in the main queue to avoid blocking AppKit app life-cycle delegate methods.
// Don't call this code in the production environment.
DispatchQueue.main.async {
    do {
        logDebug("initializeCloudKitSchema")
        try container.initializeCloudKitSchema()
    } catch {
        logWarning("initializeCloudKitSchema FAILED: \(error.localizedDescription)")
    }
}
#endif

// Pin view context of the container to the current generation token.
// This ensures that the container's view context fetches data based on the latest
// data available in the CloudKit database, thus maintaining consistency between the
// local Core Data store and the CloudKit data.
do {
    try container.viewContext.setQueryGenerationFrom(.current)
} catch {
    fatalError("Failed to pin viewContext to the current generation:\(error)")
}

⚠️ The container.initializeCloudKitSchema threw me off a bit, without this the development version of my app was working just fine from Xcode. Only when I published the app to the App Store, things started breaking during the sync. Just letting CloudKit automatically create the schema and then promoting this schema to Production was not enough. It is discussed here.

Reducing memory usage while importing

I wanted to test with a lot of data in the app. For this, I used the Import functionality that I already built in previous versions. But as it turned out, that functionality had a bug in it. It would keep all entities in memory, even though I was calling NSManagedObjectContext.save() after each imported entity. This is a similar situation to what I encountered during the the 1.7 release.

The solution to this first problem was easy (in hindsight): use autoreleasepool in the loop like this:

for index in stride(from: 0, to: importedBeers.count, by: 1) {
    autoreleasepool {
        guard let importedBeer = importedBeers[safe: index] else {
            return
        }

        // Import the beer
        ...

        // Save context
        context.saveIfChanged()
        
        // Reset context, this causes all objects to be faulted again
        context.reset()
    }
}

This massively improved the memory usage. Also the NSManagedObjectContext.reset() call improved things a bit more.

However, CloudKit does not seem to like it much if you add a lot of data at once this way. The behaviour I observed was that it would add entities just fine but at some point it would cause a massive memory spike (2.77GB!):

In a simulator this is not much of a problem, but on my actual iPhone 11 Pro it would quickly consume so much memory (2GB) that the app was killed by the OS.

I experimented a lot and in the end I figured out it was (unfortunately) necessary to give CloudKit some time to sync in the background. Even though this slowed down the import process, it made the app much more reliable. Import is done on a background context so the sleep does not block the main UI thread.

I ended up with these values: sleep 10 seconds every 50 imported Beer entities. Sleeping for less time made for more unreliable behaviour. This was the resulting graph:

No matter how much entities I tried to import, I could not get memory usage to be above 500MB so that was acceptable. It feels really hacky to me though.

So let’s look at the complete solution. First, here are the constants:

/// Save  and reset child NSManagedObjectContext every x objects while importing / adding sample data
let importSaveInterval = 50

/// How many seconds to sleep per importSaveInterval
let importSaveTimeInterval: TimeInterval = 10

and here is the complete loop:

for index in stride(from: 0, to: importedBeers.count, by: 1) {
    autoreleasepool {
        guard let importedBeer = importedBeers[safe: index] else {
            return
        }

        // Import the beer
        ...

        // Intermediate save
        if index > 0 && index.isMultiple(of: importSaveInterval) {
            logDebug("Saving context at index=\(index)")

            // Save and reset context
            context.saveIfChanged()
            context.reset()

            // Give CloudKit some time to catch up
            Thread.sleep(forTimeInterval: importSaveTimeInterval)
        }
    }
}

// Final save
logDebug("Saving context final")
context.saveIfChanged()

I spent a huge amount of time on this so it will have to do for now. But in future, I would love to solve this in a cleaner way.

Deploying schema changes to production

As mentioned before, just letting CloudKit create the schema definitions on the fly and promoting them to Production does not seem to be enough. I really needed to call container.initializeCloudKitSchema at least once in my Development environment (and then it could be promoted to Production).

Below is a partial diff of my scheme. On the left is the auto-created scheme, on the right the properly initialized scheme:

Quite a bit was missing it seems. This was definitely a learning moment!

Monitoring status

CloudKit itself is pretty much a black box. It basically either works or it does not. And if it does not, there are some statuses you can collect but they are spread over multiple information sources.

Luckily Grant Grueninger made the excellent CloudKitSyncMonitor. It is easy to hook up from SwiftUI using one single @ObservedObject. It even provides you with suggestions for icons and icon colors for each possible state so it is really easy to get started. I have chosen to filter which states are presented to the user to prevent information overload (except in Debug builds). So far it has proven to be pretty reliable.

Summary

This is a short summary on what you need to do to get up and running with CloudKit in CoreData. For the details, refer to the content above.

1. Configure your project in the correct way:
   • Add ‘iCloud’ and ‘Push Notifications’ to the app ID configuration.
   • Use NSPersistentCloudKitContainer.
   • In Xcode, enable ‘Remote notifications’ background mode.
   • Also add iCloud capability and select CloudKit.
   • Then configure the correct container identifier in the CloudKit options.
2. Configure your NSPersistentCloudKitContainer in the correct way:
   • Specify the containerIdentifier or else sync won’t start!
   • Enable persistent history tracking (NSPersistentHistoryTrackingKey).
   • Register for remote change notifications (NSPersistentStoreRemoteChangeNotificationPostOptionKey).
   • Pin the query generation (container.viewContext.setQueryGenerationFrom(.current)).
   • Configure the main view context to automatically merge changes from the parent (container.viewContext.automaticallyMergesChangesFromParent = true)
   • In your DEBUG build, call container.initializeCloudKitSchema.
3. Be careful when importing a lot of data at once:
   • Import on a separate background NSManagedObjectContext.
   • Wrap the import loop in autoreleasepool to prevent keeping all data in memory at once.
   • Call NSManagedObjectContext.save() and NSManagedObjectContext.reset() every X amount of entities.
   • Sleep every X amount of entities to give CloudKit a chance to sync some state. If you don’t add this, it seems to cause massive memory spikes that might potentially get your app killed.
4. Releasing the app:
   • Just having CloudKit creating the entities on the fly while developing and then promoting this scheme to Production in CloudKit console is not enough.
   • Make sure to publish any changes from Development to Production that are the result from calling container.initializeCloudKitSchema.

In the end I feel really proud for having navigated all these issues but it could have been a lot easier if the documentation was a bit more centralized. This blog post aims to do exactly that, if you have any remarks or additions please let me know and I will update it.