{"id":72978,"date":"2025-02-07T05:22:14","date_gmt":"2025-02-07T05:22:14","guid":{"rendered":"https:\/\/peraltafinancing.com\/apple-2\/follow-on-to-dfir-summit-talk-lucky-ios-13-time-to-press-your-bets-via-bizzybarney\/"},"modified":"2025-02-07T05:22:14","modified_gmt":"2025-02-07T05:22:14","slug":"follow-on-to-dfir-summit-talk-lucky-ios-13-time-to-press-your-bets-via-bizzybarney","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=72978","title":{"rendered":"Follow-on to DFIR Summit Talk: Lucky (iOS) 13: Time To Press Your Bets (via @bizzybarney)"},"content":{"rendered":"


\n<\/p>\n

Facial Recognition in Photos<\/h2>\n

One facet of my DFIR Summit talk I want to expand upon is a look into the Photos application, and a few of the derivative pieces of that endeavor.  While trying to focus on the topic of facial recognition, it seemed prudent to include a brief progression from snapping a photo thru to a persons name being placed beside their face in the Photos application.  <\/p>\n

When you use the Native camera and snap a photo, depending on user options, at least a few standard things occur.  It ultimately writes the newly taken photo to \/private\/var\/mobile\/Media\/DCIM\/1**APPLE\/IMG_0001.HEIC \/ .JPG.  As the photo is taken, the Photos.sqlite database is updated to reflect a lot of the metadata about the photo,  which will be covered a bit later.  Additionally, the \u201cPreviewWellImage.tiff\u201d is created.  The \u201cPreviewWellImage.tiff\u201d represents the photo you see when you open your Photos application and see a preview of the most recent image, which is the photo just taken by the camera in this instance.<\/p>\n

<\/p>\n

\"\"<\/p>\n<\/figure>\n

The beginning of the user\u2019s photos reside in the ..\/100APPLE\/ directory, but this directory iterates upwards (101APPLE, 102APPLE, etc) as more and more photos and videos are saved.  If iCloud syncing is turned on by the user, then several others behaviors occur – but that is for another time.<\/p>\n

Let\u2019s focus on the analysis and intelligence built into the Photos application.  I\u2019m able to type text strings and my photos are immediately searched for matching objects within them.  There is a section of my Photos dedicated to \u201cPeople\u201d where a name has been associated with a face that Apple has analyzed.  <\/p>\n

<\/p>\n

\"\"<\/p>\n<\/figure>\n

<\/p>\n

\"\"<\/p>\n<\/figure>\n

Some of the analysis pieces occurring with the Photos happens in the mediaanalysis.db file.  This file is analyzing and scoring the media files and producing results that seems to feed into other pieces of the analysis.  Some scoring results to highlight are ones that focus on Humans, Faces, and Pets.  <\/p>\n

Path: \/private\/var\/mobile\/Media\/MediaAnalysis\/mediaanalysis.db<\/p>\n

The \u2018Results\u2019 table of the mediaanalysis.db file contains the \u2018assetId\u2019 which represents a media file, a \u2018resultsType\u2019 which is the specific analytical type and score value found in that media file, and a BLOB (binary large object) which is a binary .plist (bplist).  You can see in the image below, the \u2018assetId\u2019 3 has numerous \u2018resultsType\u2019 associated with it and the BLOB for \u2018resultsType\u2019 of 1 is selected and on the right you can see the bplist.  <\/p>\n

<\/p>\n

\"\"<\/p>\n<\/figure>\n

That bplist can be printed to a text file by saving the bplist as a file and then using \u2018plutil\u2019 to print it to a text file. As you can see below beside the red star, the printed text is a clean presentation of that .bplist and it tells us that \u2018resultsType\u2019 of 1 is associated with Faces based on the scoring.  <\/p>\n

<\/p>\n

\"\"<\/p>\n<\/figure>\n

I repeated that process for the remaining pieces of data and wrote a brief description of the results type for each, although my device still had a few types that did not have any results.<\/p>\n

<\/p>\n

\"\"<\/p>\n<\/figure>\n

<\/p>\n

\"\"<\/p>\n<\/figure>\n

After running a SQL Query against this database, you can sort the results to potentially see just files that have the results type for \u2018humanBounds\u2019 and \u2018humanConfidence\u2019.  The \u2018localIdentifier\u2019 column from the \u201cassets\u2019 table is a UUID which matches up to the ZGENERICASSETS table of the Photos.sqlite.  <\/p>\n

Here is the query for the mediaanalysis.db file.  It\u2019s big and ugly but please test it out if you\u2019re interested, but this piece just seems to build into what we will see later in the Photos.sqlite where this all comes together.<\/p>\n

select<\/code><\/pre>\n
\ta.id,<\/code><\/pre>\n
\ta.localIdentifier as \"Local Identifier\",<\/code><\/pre>\n
\ta.analysisTypes as \"Analysis Types\",<\/code><\/pre>\n
\tdatetime(a.dateModified+978307200, 'unixepoch') as \"Date Modified (UTC)\",<\/code><\/pre>\n
\tdatetime(a.dateAnalyzed+978307200, 'unixepoch') as \"Date Analyzed (UTC)\",<\/code><\/pre>\n
CASE<\/code><\/pre>\n
\twhen results.resultsType = 1 then \"Face Bounds \/ Position \/ Quality\"<\/code><\/pre>\n
\twhen results.resultsType = 2 then \"Shot Type\"<\/code><\/pre>\n
\twhen results.resultsType = 3 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 4 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 5 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 6 then \"Duration \/ Flags \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 7 then \"Duration \/ Flags \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 15 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 19 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 22 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 23 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 24 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 25 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 27 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 36 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 37 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 38 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 39 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 48 then \"Duration \/ Quality \/ Start\"<\/code><\/pre>\n
\twhen results.resultsType = 8 then \"UNK\"<\/code><\/pre>\n
\twhen results.resultsType = 11 then \"UNK\"<\/code><\/pre>\n
\twhen results.resultsType = 13 then \"UNK\"<\/code><\/pre>\n
 \twhen results.resultsType = 21 then \"UNK\u201d <\/code><\/pre>\n
\twhen results.resultsType = 26 then \"UNK\"<\/code><\/pre>\n
\twhen results.resultsType = 31 then \"UNK\"<\/code><\/pre>\n
\twhen results.resultsType = 42 then \"UNK\"<\/code><\/pre>\n
\twhen results.resultsType = 45 then \"UNK\"<\/code><\/pre>\n
\twhen results.resultsType = 49 then \"UNK\"<\/code><\/pre>\n
\twhen results.resultsType = 9 then \"Attributes - junk\"<\/code><\/pre>\n
\twhen results.resultsType = 10 then 'Attributes - sharpness'<\/code><\/pre>\n
\twhen results.resultsType = 12 then \"Attributes - featureVector\"<\/code><\/pre>\n
\twhen results.resultsType = 14 then \"Attributes - Data\"<\/code><\/pre>\n
\twhen results.resultsType = 16 then \"Attributes - orientation\"<\/code><\/pre>\n
\twhen results.resultsType = 17 then 'Quality'<\/code><\/pre>\n
\twhen results.resultsType = 18 then \"Attributes - objectBounds\"<\/code><\/pre>\n
\twhen results.resultsType = 20 then \"Saliency Bounds and Confidence\"<\/code><\/pre>\n
\twhen results.resultsType = 28 then \"Attributes - faceId \/ facePrint\"<\/code><\/pre>\n
\twhen results.resultsType = 29 then \"Attributes - petsBounds and Confidence\"<\/code><\/pre>\n
\twhen results.resultsType = 30 then \"Various Scoring Values\"<\/code><\/pre>\n
\twhen results.resultsType = 32 then \"Attributes - bestPlaybackCrop\"<\/code><\/pre>\n
\twhen results.resultsType = 33 then \"Attributes - keyFrameScore \/ keyFrameTime\"<\/code><\/pre>\n
\twhen results.resultsType = 34 then \"Attributes - underExpose\"<\/code><\/pre>\n
\twhen results.resultsType = 35 then \"Attributes - longExposureSuggestionState \/ loopSuggestionState\"<\/code><\/pre>\n
\twhen results.resultsType = 40 then \"Attributes - petBounds and Confidence\"<\/code><\/pre>\n
\twhen results.resultsType = 41 then \"Attributes - humanBounds and Confidence\"<\/code><\/pre>\n
\twhen results.resultsType = 43 then \"Attributes - absoluteScore\/ humanScore\/ relativeScore\"<\/code><\/pre>\n
\twhen results.resultsType = 44 then \"Attributes - energyValues\/ peakValues\"<\/code><\/pre>\n
\twhen results.resultsType = 46 then \"Attributes - sceneprint\/ EspressoModelImagePrint\"<\/code><\/pre>\n
\twhen results.resultsType = 47 then \"Attributes - flashFired, sharpness, stillTime, texture\"<\/code><\/pre>\n
end as \"Results Type\",<\/code><\/pre>\n
\thex(results.results) as \"Results BLOB\"<\/code><\/pre>\n
from assets a<\/code><\/pre>\n
left join results on results.assetId=a.id<\/code><\/pre>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
Before diving into the Photos.sqlite file, I want to first point out the file recording the text strings as an apparent result of Apple\u2019s object analysis of the photos.  This file stores the text results which empower our ability to search text strings in the Photos application and return results.  The text strings are not necessarily a result of any specific user activity, but instead an output from analysis automatically being deployed by Apple against the user\u2019s media files.  <\/p>\n
Path: \/private\/var\/mobile\/Media\/PhotoData\/Caches\/search\/psi.sqlite<\/p>\n
The \u2018word_embedding\u2019 table within psi.sqlite contains columns \u2018word\u2019 and \u2018extended_word\u2019 which are just strings stored in BLOB\u2019s.  Using DB Browser for SQLite you can export the table to a CSV and it prints the strings from the BLOB\u2019s pretty cleanly.  Separately there is also a table named \u2018collections\u2019 that has \u2018title\u2019 and \u2018subtitle\u2019 columns that appear to be a history of the Memories and Categories that have been used or ones that will be used.<\/p>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
The last table in psi.sqlite to mention for this piece is the \u2018groups\u2019 table.  Within the groups table the \u2018content_string\u2019 contains some really interesting data.  I initially set out to find just the words \u201cGreen Bay\u201d as it was something populated in my text search for the letter \u201cg\u201d.  What I found was far more interesting.  I did find \u201cGreen Bay\u201d but additionally found in one of the other BLOB\u2019s, \u201cGreen Bay Packers vs. Miami Dolphins\u201d.  That BLOB has a little extra flavor added by Apple for me.  Whether they simply used my geo coordinates baked into my photos from being at Lambeau Field, or analyzed the content of the photos and found the various Miami Dolphins jerseys – I\u2019m not sure.  But a very interesting artifact that is absolutely accurate, dropped in there for me.  Thanks Apple!<\/p>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
Now let\u2019s tackle Photos.sqlite, but only as it pertains to facial recognition and associating photos of people to an actual name.  Because quite honestly this singular file is nearly a full time job if someone wanted to parse every inch of it, and maintain that support.<\/p>\n
Path: \/private\/var\/mobile\/Media\/PhotoData\/Photos.sqlite<\/p>\n
My instance of Photos.sqlite is a beast, weighing in at over 300MB and containing 67 tables packed full of data about my Photos.  We are going to focus on two tables – ZDETECTEDFACE and ZPERSON.  <\/p>\n
ZDETECTEDFACE <\/h3>\n
This table contains values that indicate features about faces to include an estimate of age, hair color, baldness, gender, eye glasses, and facial hair.  Additionally there are indicators for if the left or right eyes were closed, and X and Y axis measurements for the left eye, right eye, mouth and center.  So the data in this table is extremely granular, and was quite fun to work through.  Who doesn\u2019t like looking at old photos?<\/p>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
ZPERSON<\/h3>\n
This table contains a count for the number of times Apple has been able to identify a certain face from the media files.  So in my device, I am recognized by name for hundreds of photos, but there are also photos of me where it hasn\u2019t associated my name with my face.   For each face identified, a UUID (Unique Identifier) is assigned.  So although the analytics piece may not be able to connect a face with a name, it can group all identified instances of the unknown faces as being the same person. <\/p>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
If there is an association made between the person\u2019s face and a saved contact, the ZCONTACTMATCHINGDICTIONARY column\u2019s BLOB data can possibly reveal a full name and the phone number.  This again can be achieved by printing the bplist to a .txt file.<\/p>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
select<\/code><\/pre>\n
\tzga.z_pk,<\/code><\/pre>\n
\tzga.ZDIRECTORY as \"Directory\",<\/code><\/pre>\n
\tzga.ZFILENAME as \"File Name\",<\/code><\/pre>\n
CASE<\/code><\/pre>\n
\twhen zga.ZFACEAREAPOINTS > 0 then \"Yes\"<\/code><\/pre>\n
\telse \"N\/A\"<\/code><\/pre>\n
\tend as \"Face Detected in Photo\",<\/code><\/pre>\n
CASE <\/code><\/pre>\n
\twhen zdf.ZAGETYPE = 1 then \"Baby \/ Toddler\"<\/code><\/pre>\n
\twhen zdf.ZAGETYPE = 2 then \"Baby \/ Toddler\"<\/code><\/pre>\n
\twhen zdf.ZAGETYPE = 3 then \"Child \/ Young Adult\"<\/code><\/pre>\n
\twhen zdf.ZAGETYPE = 4 then \"Young Adult \/ Adult\"<\/code><\/pre>\n
\twhen zdf.ZAGETYPE = 5 then \"Adult\"<\/code><\/pre>\n
end as \"Age Type Estimate\",<\/code><\/pre>\n
case<\/code><\/pre>\n
\twhen zdf.ZGENDERTYPE = 1 then \"Male\"<\/code><\/pre>\n
\twhen zdf.ZGENDERTYPE = 2 then \"Female\"<\/code><\/pre>\n
\telse \"UNK\"<\/code><\/pre>\n
end as \"Gender\",<\/code><\/pre>\n
\tzp.ZDISPLAYNAME as \"Display Name\", <\/code><\/pre>\n
\tzp.ZFULLNAME as \"Full Name\",<\/code><\/pre>\n
\tzp.ZFACECOUNT as \"Face Count\",<\/code><\/pre>\n
CASE\t<\/code><\/pre>\n
\twhen zdf.ZGLASSESTYPE = 3 then \"None\"<\/code><\/pre>\n
\twhen zdf.ZGLASSESTYPE = 2 then \"Sun\"<\/code><\/pre>\n
\twhen zdf.ZGLASSESTYPE = 1 then \"Eye\"<\/code><\/pre>\n
\telse \"UNK\"<\/code><\/pre>\n
end as \"Glasses Type\",<\/code><\/pre>\n
CASE<\/code><\/pre>\n
\twhen zdf.ZFACIALHAIRTYPE = 1 then \"None\"<\/code><\/pre>\n
\twhen zdf.ZFACIALHAIRTYPE = 2 then \"Beard \/ Mustache\"<\/code><\/pre>\n
\twhen zdf.ZFACIALHAIRTYPE = 3 then \"Goatee\"<\/code><\/pre>\n
\twhen zdf.ZFACIALHAIRTYPE = 5 then \"Stubble\"<\/code><\/pre>\n
\telse \"UNK\"<\/code><\/pre>\n
end as \"Facial Hair Type\",<\/code><\/pre>\n
CASE\t<\/code><\/pre>\n
\twhen zdf.ZBALDTYPE = 2 then \"Bald\"<\/code><\/pre>\n
\twhen zdf.ZBALDTYPE = 3 then \"Not Bald\"<\/code><\/pre>\n
end as \"Baldness\",<\/code><\/pre>\n
CASE<\/code><\/pre>\n
\twhen zga.zlatitude = -180<\/code><\/pre>\n
\tthen 'N\/A'<\/code><\/pre>\n
\telse zga.ZLATITUDE<\/code><\/pre>\n
end as \"Latitude\",<\/code><\/pre>\n
CASE<\/code><\/pre>\n
\twhen zga.ZLONGITUDE = -180 <\/code><\/pre>\n
\tthen 'N\/A' <\/code><\/pre>\n
\telse zga.ZLONGITUDE<\/code><\/pre>\n
end as \"Longitude\",<\/code><\/pre>\n
\tdatetime(zga.zaddeddate+978307200, 'unixepoch') as \"Date Added (UTC)\",<\/code><\/pre>\n
\tZMOMENT.ztitle as \"Location Title\"<\/code><\/pre>\n
from zgenericasset zga<\/code><\/pre>\n
left join zmoment on zmoment.Z_PK=zga.ZMOMENT<\/code><\/pre>\n
left join ZDETECTEDFACE zdf on zdf.ZASSET=zga.Z_PK<\/code><\/pre>\n
left join ZPERSON zp on zp.Z_PK=zdf.ZPERSON<\/code><\/pre>\n
where zga.ZFACEAREAPOINTS > 0<\/code><\/pre>\n
Below is a sample of the output of this analysis, paired with the photo the metadata came from.  You can see it is able to identify me and my two daughters by name, and accurately assess our genders, my sunglasses and facial hair.  <\/p>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
<\/p>\n
                \"\"<\/p>\n<\/figure>\n
Please test, verify, and give me a shout on Twitter @bizzybarney<\/a> with any questions or concerns.  <\/p>\n