Beer Diary version 1.6.1 released

Version 1.6.1 of my BeerDiary app is now live on the iOS AppStore! You can download it here.

I would love to receive your feedback. If you have questions or remarks don’t hesitate to send me a message.

New features

Data migration

No user-facing new features are present in this version. However, you will briefly see a see a see a migration screen when the app is opened for the first time:

Depending on the number of beers you have in the app and the speed of your device, this process will take anywhere from a few seconds to up to a minute. Just leave the app in the foreground and it will finish automatically. This process will only run once.

⚠️ Due to technical reasons (refer to the ‘Implementation notes’ section if you’re really curious) the order in which photos are shown in the beer detail screen might have changed. Please drag and drop them back to your preferred order. I’m sorry for any inconvenience this might cause and I will do my best to prevent this in future.

Analytics

To better understand how users are using the app, I added anonymized analytics using TelemetryDeck. This was the most privacy-friendly analytics package I could find and it is configured to send as little information as possible. You can read more about TelemetryDeck’s privacy policy here.

Implementation notes

Feel free to skip this section if you’re not that technical 😁

No more MVVM

Last release I wrote why I started migrating away from MVVM in this app. You can read my motivation in the ‘Implementation notes’ section of the release post. For this release, I finished the process and removed all remaining View Models and moved everything back into the actual Views.

For some cases, I ended up with huge Views with a lot of code and @State properties. This is exactly what MVVM was made to solve right? But actually working with SwiftUI without the MVVM layer in between made the solution quite obvious: instead of having one large View, subdivide it into smaller component Views that each have their own relevant state. This was quite a bit of work but the actual components became much simpler and more reusable.

Of course, I could have used the same approach with MVVM but the hassle of adding a View Model for each component View quickly makes it feel like a lot of work. Staying closer to SwiftUI really felt like the proper solution for this project. It allowed me to quickly extract component views with minimum hassle. I can imagine for larger projects, MVVM would still be a good choice though.

In future, I will refactor a bit more to MV architecture and start with some high-level services that are injected via the @Environment instead of using singletons.

CoreData migration

In an upcoming version, I want to support CloudKit. But I’m using an ordered relationship between a Beer and the Photo entities that belong to each instance. This is not supported by CloudKit so I had to migrate away from this.

The simple solution is to add an additional userOrder property to Photo and use that to control the order in which all photos are shown. When you drag/drop to update the order, the properties for all affected photos are manually updated in code. It was a bit of experimentation to get it right but in the end I’m satisfied with the solution and it even allowed me to remove some less-pretty code.

However to update to a new schema version, CoreData needs to perform a migration. There are a number of migration types possible:

1. Lightweight. CoreData automatically infers the changes and performs the migration automatically. This is done on SQLite level and is pretty efficient and quick.
2. Heavyweight with mapping model. If the changes are too big for CoreData to automatically deduce the changes, you create a mapping model in Xcode to help make the right changes.
3. Heavyweight with NSEntityMigrationPolicy. This scenario is recommended if you need to do additional processing that a mapping model can’t handle.

To evaluate these solutions, I loaded up a BeerDiary app with a 1000 sample beers. This is a huge amount but I wanted to get a feeling for the CPU/memory cost of each scenario.

Lightweight

Initially I tried to use a lightweight migration but the downside to that is that the ordered relationship automatically gets converted to an unordered relationship and therefore any custom photo sorting order made by the user is lost. It can be an option to just add a second (unordered) relationship and migrate manually. Then in a future version, remove the ordered relationship. So the data model versions will look like this:

• V1: ordered relationship Beer->Photo
• V2: ordered relationship Beer->Photo, unordered relationship Beer->Photo and add userOrder to Photo
• V3: remove ordered relationship Beer->Photo after migration from V1->V2

The problem here is that it is theoretically possible that a user still has an old version installed with the V1 model and will need to do a V1->V3 migration. In that case data is still lost. And this refactor would then have to be spread out over 2 app versions, which is less than optimal.

Heavyweight

I tried looking into a mapping model but this would not work for me. Basically, for each Beer I would have to go over each linked Photo in order and update the userOrder accordingly. Maybe it’s possible with a mapping model but I did not investigate further.

So next I tried the NSEntityMigrationPolicy. It was a bit of a learning journey because it seems to be not that well documented. In the end I made it work, for future reference here is my version:

import Foundation
import CoreData
import UIKit

class V1toV2MigrationPolicy: NSEntityMigrationPolicy {

    /// Migrate Beer instance
    /// - Parameters:
    ///   - sourceBeer: Source instance
    ///   - mapping: Entity mapping
    ///   - manager: Migration manager
    override func createDestinationInstances(forSource sourceBeer: NSManagedObject,
                                             in mapping: NSEntityMapping,
                                             manager: NSMigrationManager) throws {
        guard let destinationEntityName = mapping.destinationEntityName else {
            logWarning("Could not get destinationEntityName")
            return
        }

        guard destinationEntityName == "Beer" else {
            logWarning("Policy expects Beer, got \(destinationEntityName) instead")
            return
        }

        logDebug("Start migration of Beer: \(sourceBeer.value(forKey: "name") ?? "nil")")

        // Migrate all keys from source to destination
        let destinationBeer = migrateAllKeys(in: sourceBeer, entityName: destinationEntityName, manager: manager)

        //  Handle the relationship to Photo entities, preserving the original order
        migratePhotos(in: sourceBeer, to: destinationBeer, manager: manager)

        logDebug("Finished migration of Beer: \(sourceBeer.value(forKey: "name") ?? "nil")")
    }

    // MARK: - Private

    /// Migrate all Photo instances and convert ordering to new situation
    /// - Parameters:
    ///   - sourceBeer: Source Beer instance
    ///   - destinationBeer: Destination Beer instance
    ///   - manager: Migration manager
    private func migratePhotos(in sourceBeer: NSManagedObject,
                               to destinationBeer: NSManagedObject,
                               manager: NSMigrationManager) {

        guard let sourcePhotoSet = sourceBeer.value(forKey: "photos") as? NSOrderedSet else {
            logWarning("Expected 'photos' to be an NSOrderedSet, cannot proceed!")
            return
        }

        guard let sourcePhotos = sourcePhotoSet.array as? [NSManagedObject] else {
            logWarning("Could not convert to array, cannot proceed!")
            return
        }

        logDebug("Beer has \(sourcePhotos.count) Photos")
        var userOrder = 0
        for sourcePhoto in sourcePhotos {
            logDebug("Migrate Photo \(userOrder): \(sourcePhoto.value(forKey: "timestamp") ?? "nil")")

            // Migrate all keys from source to destination
            let destinationPhoto = migrateAllKeys(in: sourcePhoto, entityName: "Photo", manager: manager)

            // Set new userOrder field
            logDebug("photo.userOrder=\(userOrder)")
            destinationPhoto.setValue(userOrder, forKey: "userOrder")

            // Update Beer thumbnail to the first image
            if userOrder == 0 {
                logDebug("Updating Beer thumbnail")
                let thumbnailData = sourcePhoto.value(forKeyPath: #keyPath(Photo.thumbnailData))
                destinationBeer.setValue(thumbnailData, forKeyPath: #keyPath(Beer.thumbnailData))
            }

            userOrder += 1

            // Add the Photo entity to the Beer's "photos" relationship
            let photosSet = destinationBeer.mutableSetValue(forKey: "photos")
            photosSet.add(destinationPhoto)
        }
    }

    /// Migrate all keys in source object
    /// - Parameters:
    ///   - source: Source instance
    ///   - entityName: Entity name
    ///   - manager: Migration manager
    /// - Returns: Migrated instance containing source keys (if still available in this new version)
    private func migrateAllKeys(in source: NSManagedObject,
                                entityName: String,
                                manager: NSMigrationManager) -> NSManagedObject {
        // Examine source
        let sourceKeys = source.entity.attributesByName.keys
        let sourceValues = source.dictionaryWithValues(forKeys: sourceKeys.map { $0 as String })

        // Examine destination
        let destination = NSEntityDescription.insertNewObject(forEntityName: entityName,
                                                              into: manager.destinationContext)
        let destinationKeys = destination.entity.attributesByName.keys.map { $0 as String }

        // Copy values
        for key in destinationKeys {
            if let value = sourceValues[key] {
                destination.setValue(value, forKey: key)
                logDebug("\(entityName).\(key)=\(value)")
            }
        }

        return destination
    }
}

However, when you run this solution on the 1000 beers scenario it quickly ran out of memory since I’m storing all images inside CoreData itself. For reference, with 1000 beers and 1150 photos (some had 4 photos to check the order) I ended up with a SQLite store of about 3.6gb. The memory usage in the simulator was about equal to that, on an actual device I hit the 2gb RAM memory limit pretty quickly and my app was killed.

For some reason, the migration seemed to hold on to each source object loaded into memory. Things I tried to mitigate this:

1. Regularly save the destination NSManagedObjectContext. This made almost no difference.
2. Re-fault objects after they were done migrating and saving the destination context. This helped a little but not enough. For future reference: re-faulting an object can be done like this:

if let context = sourcePhoto.managedObjectContext {
    logDebug("Refresh source photo")
    context.refresh(sourcePhoto, mergeChanges: false)
}

So while this policy would in theory be the ideal solution, it would not be realistic for users with a huge data set. I could have been pragmatic and done it anyway (because who has a 1000 beers in the app?) but that felt like cheating. Also this step would take quite a long time, about 1 to 2 minutes.

Thinking outside the box

As a side step, I also tried to create another data store with the new model and just manually copy all objects to the new store. The copying step was quite easy to build and pretty quick, but then came the real problem: SwiftUI still had references to the old objects. How do I refresh the UI to use the new store? I did not find a way. There probably is a solution but I could not find it. It would work fine after restarting the app but that was not a very good user experience.

Choosing lightweight

So in the end, I spent a couple of days trying different ways to optimize the heavyweight migration but I could not get it to work. Maybe there is a way but I ran out of time and energy to find it. So I went for the lightweight migration. I benchmarked some different scenarios (adding another relationship, converting the existing one, adding userOrder or not adding it at all and re-using some other field I had not used yet in Photo). After comparing there was little difference time-wise.

The lightweight solution consumed about 50mb of RAM for the actual migration (since it is done on SQLite level) and then about 200mb for a manual fix-up step. And it finishes in about 30 seconds. Because all ordering was lost, I felt it was necessary to at least update all userOrder fields of linked Photo entities to something else than their default value. This way, any future changes in sorting logic further on would not cause another re-order for the user. I used parts of the logic from the heavyweight migration policy for this.

So I wrote a method that is performed only once and the most important thing about this it is that it is running on a background NSManagedObjectContext so the app should be usable in the mean time. The only thing a user might notice in the list view is that a thumbnail for a beer might change (due to the new ordering criteria) but that was acceptable to me. The method loops over all Beer instances and writes the userOrder to the linked Photo instances according to the new sort order.

Blocking the main thread

My app is using NSPersistentContainer to manage the CoreData store. You do a bit of set up and at some point you call container.loadPersistentStores. A nasty surprise here is that this call will block your main thread until the migration is done. This means all UI interaction with your app is disabled. I “solved” this by detecting if a migration will be performed beforehand and showing a launch screen with a little infinite spinning ProgressView. For some reason, the spinner would still animate so this was good enough for now. At some point you also have to say ‘enough is enough, let’s be pragmatic’.

Again, there are probably solutions for this. If you have one, let me know and I will add it to this page 😀.

Storage requirements

One downside of performing a migration seems to be the CoreData basically duplicates the existing store into the new store. This results in a SQLite store that doubles in size. During subsequent usage of the app a process called ‘auto vacuum’ will be performed in steps and at some point the store size will be back to the original size. But this vacuum process has a habit of blocking the main thread it seems. It’s only for short moments but it feels a bit messy and unpredictable.

Conclusion and lessons learned

So I unfortunately ended up throwing away any custom sorting order the user might have made for their photos. Sorry! This was a conscious decision, but one that I did not take lightly. The alternative was potentially getting users stuck in a long-running migration step that might run out of memory. Which is even worse in my opinion. Balancing these pros and cons is a hard part of software development but I’m convinced I made the right choice.

So there are some takeaway points for me that I would like to share:

⚠️ Lesson 1: CloudKit does not support all features of a CoreData model. Refer to this article from Apple to learn more about them. If you are ever even considering adding CloudKit support in the future, design your data model with these limitations in mind!

⚠️ Lesson 2: If you need a migration, try to get away with a lightweight version. It is much more memory efficient and quicker to execute.

⚠️ Lesson 3: Be careful that all migration types end up blocking your main thread when using NSPersistentContainer. There probably is a way around this but I did not find one yet.

⚠️ Lesson 4: Be aware that storage requirements will double during a migration. And that your app will auto-vacuum itself back to the original size, but it might shortly block the main thread.

Hopefully I will not need to migrate much in future but it was interesting learning about this for sure. Did I miss any obvious solutions? Any feedback is always welcome!

Analytics

I’m pretty skeptical about using any analytics solution. Privacy is something I value a lot. However, when considering the CoreData migration scenarios I ended up with one conclusion: I don’t know much about how my users are using the app. Up until this version, I only had some basic figures from Sentry but that is mostly focussed on crashes.

So I bit the bullet and added the most privacy-friendly framework I could find: TelemetryDeck. It’s free up to 100k signals so this should be more than enough for me. There is no way for me to link signals to a specific device or user, everything depends on an anonymized ID. You can read more about TelemetryDeck’s privacy policy here. Also, the app privacy report section in the App Store has been updated so you know exactly what is being shared.

I’ve added some basic usage signals, including how long it takes to load the persistent store and migration duration. No actual user data will ever be shared, you have my pinky promise for this. It’s just so I have a idea how the app is used so I can focus on those areas.