Home / Engineering notes / Data map
Data Map engineering notes
Lessons and tool behaviour found while building the Live Data Map.
Lessons and tool behaviour found while building the Live Data Map (ADR-157, ADR-158): the collector in system_server, the classifier file and DataMapApp. The design is 2026-10-11-live-data-map-design.md; the patches, their baselines, the landing and the module build are recorded in datamap/SOURCE.md of the oblivios-platform repository. Claims are labelled [verified-in-source] (the path is named) or [measured] (the run is named).
1The netd callback thread must not block
NetdEventListenerService.onDnsEvent and onConnectEvent are synchronized and call each registered callback with that monitor held, on a binder thread of the resolver; their own comments say they "must not block or perform long-running operations" [verified-in-source: frameworks/base services/core/java/com/android/server/connectivity/NetdEventListenerService.java]. The first version of the collector asked the activity manager (getUidState takes mProcLock, [verified-in-source: .../am/ActivityManagerService.java]) and the package manager from that thread, so a slow lock would have stalled every consumer of the events. The collector now validates the name and hands the event to the service thread through a bounded counter (PendingLimit, 4096); an event that finds no place is dropped and counted.
Unrelated to the thread, but easy to misread: the callback runs inside a transaction from netd, which is root, so Binder.getCallingUid() there is root and the package filter lets every package through [verified-in-source: .../pm/AppsFilterBase.java]. Attribution was right by accident of that uid, not by design.
2Order of events at user stop
UserController.finishUserStopped calls onUserStopped and then, synchronously, locks the user's credential-encrypted storage [verified-in-source: .../am/UserController.java]. A write only posted from onUserStopped runs after the lock and is lost. The final write belongs in onUserStopping, and a receiver for ACTION_SHUTDOWN writes at shutdown.
3What a type-check cannot see: the first module build
A type-check of the new classes against the jars of an earlier build passed, and the first m of the modules still failed twice:
fuzzer_bindings_testrefuses a new service that is not listed insystem/sepolicy/build/soong/service_fuzzer_bindings.go. A Java service has no automatic fuzzer (b/287102710); the exception isEXCEPTION_NO_FUZZER.- A
<permission>declared incore/res/AndroidManifest.xmlbecomes a publicManifest.permissionconstant, andcheck_current_apirefuses a public API thatcore/api/current.txtdoes not list. A permission for one app of this system needs@hidein the comment above it.
m -k shows every failure of a run; without it a run stops at the first. Soong accepted the new Android.bp files (prebuilt_etc with sub_dir and filename, android_app with AIDL) without change [measured: module builds of 2026-10-12].
4SELinux
system_appandtraceur_appare app domains with anallowfor every service type except a fixed list[verified-in-source: system/sepolicy private/system_app.te, private/traceur_app.te]. A new service type that only one app domain may find needs-datamap_servicein both lists, or theneverallowfor the other app domains does not compile.enforceCallingPermissionis satisfied by app id 1000 (the activity manager grants every permission to it), so the permission does not separate the Data Map app from the apps of the system uid; SELinux is the only gate.- A new system service needs entries in
service.te,service_contextsand, for a privileged app with its own domain,seapp_contexts(the pattern oflogviewer_app).
5Bounds of the reply
A Binder transaction is about 1 MB. A snapshot of the Data Map is bounded in the service (2000 entries, 300 links) because links are derived for every pair of apps that share a tracker and grow with the square of the apps. The history is read once per change of the file and kept for the open app only. A snapshot over the limit would arrive as a failed transaction exactly when the data is richest.
6Server places: bounds the review found
- A decoder bound that was sized for one format is not the bound of the next. The line limit of the history decoder (600) was sized for the ten-field row; a place of two names of 100 characters made a row of 634, the decoder refused the file, and the next write started from nothing. The limit is now derived from the maxima (1024) and the encoder keeps to it: a place that would pass it is left out, a row that is too long without one is not written.
- A reply is bounded by bytes, not only by rows: seven fields with names of up to 255 characters take up to 2 KB an entry.
DataMapStore.withinkeeps the newest rows that fit 600 KB, counted as the parcel holds them. - A flag added to a file that an earlier build reads must not be a new key of that file when the reader treats an unknown key as "use the defaults": a rollback would have switched collecting on. The choice to keep server places is a file of its own.
- Unsigned values in Java: AS numbers of 2**31 or more are negative
ints, and IPv6 keys with the top bit set sort first when compared as signed. A fixture that holds neither passes with the wrong compare: the fixture now has a private AS number of 4200000000 and a range ata000::. A mutation that cannot be killed is an equivalent mutant: the 48-bit anycast prefixes never have the top bit set, so the signed compare ofmember64is not a mutation to keep. Character.digitaccepts the digits of other scripts (a fullwidth1was an IPv6 digit); a parser of untrusted text names its ASCII set. A zone id belongs to an IPv6 address only;8.8.8.8%xis not an address.- Python
str.isalphatakes a letter outside ASCII, which does not fit the 16 bits of a country code:pack_cccrashed the build tool on such a row. - A test that compares a model with itself proves nothing about the wording: the check asserted that Frankfurt and Berlin are one place and never read what the card says about them. A card names the lead only, so a place that joins cities must not be named after one of them; a server place is never joined.
- A sheet is not the globe: the rule "a server place is joined to nothing" held for the lines (
PlaceLinks.pairs) and not for the list under the sheet (PlaceLinks.joined). Put a rule in the function every caller goes through. - A record that says a file is archived must name the path: the inputs of the first
ipmap.binwere said to be archived and were nowhere on the build server, so its pins could only be met by luck (thelatestrelease had not changed that day). The inputs are now kept indatamap-inputs/on the build server and checked againstPINS.sha256. - Landing a series on the build server: compare the blob of every touched file with the disposable tree after
git am(git rev-parse HEAD:<path>againstgit hash-object); the AIDL files are not in the tree thatmaterialize-core.shwrites, so compare them from the patches. A remotegrep -c '[s]oong_ui'counts its own command line when the same text is elsewhere in the command (anecho soong_ui:did): look atpsoutput, not at a count. - Launching a build over
ssh:A && B && nohup script & disownputs the whole chain in the background and keeps thesshcall open on its output. Start the script as its own command and read the log from a second call. - Merging a feature line into
mainthat has moved on (the ADR-163 squash): use a temporary worktree ofmainandgit merge --no-ff, thengit diff <branch> main -- datamapmust be empty. - A replay script of edits goes stale the moment the code is changed by hand; the two edit lists of the branch were removed, the patch and the sources are the record.
7Switch defects found on the device (2026-10-13 and 2026-10-14)
- Two Switches with one view id restored each other's value on a theme change; the interface review found it as a Critical defect and each Switch has its own id now.
- A tap on a switch toggled twice (
MAP-036): a constructor order made the Settings page set a switch before its listener existed. A page built again under a finger then lost a tap now and then, also when it was built again before its layout (captured with the touch log). Settings therefore moves its own controls in place instead of building the page again.
8The classifier data
- Sources and licences:
datamap/classifier/SOURCES.mdinoblivios-platform. The tracker lists of Disconnect, DuckDuckGo and Ghostery are CC-BY-NC-SA-4.0 and cannot ship. The AdGuard companies database is CC-BY-SA-4.0 (built on whotracks.me, CC-BY-4.0); the classifier file therefore carries CC-BY-SA-4.0 and a NOTICE. - The database has companies, domains and categories but no country. Wikidata (CC0) gives a headquarters for about 15 percent of its companies, found by the official website; Wikidata has no item with a headquarters for the other, mostly small advertising companies
[measured: tools/fetch-wikidata-bases.py, 2026-10-12]. Their services are listed with a category and no owner. The search API of Wikidata answers HTTP 429 after a few hundred requests; SPARQL does not. - The device parser caught what the builder had not: two records of the database that point to one Wikidata item gave a duplicate source, and a Wikidata item with no English label returns its code (
Q123) as label, which had become a legal name or a city in 22 owners. The importer merges the first and refuses the second. - An owner id is
[a-z0-9_]{1,24}(no hyphen).
9Process lessons
- A research agent is not a source. Every row of the hand-checked layer was read against its page by the author, and rows were dropped: a partnership whose 10-year term had ended, a category the page only implied, a domain that serves unrelated content as a whole (
unity3d.com). - Relations are shown as current. Only
ownsandmerged_withstay true once done;partner_ofandbuys_ad_space_fromneed a source that shows the arrangement is current. - A mutation count proves nothing until the unmutated tree has passed first and every mutant compiles. The app runner copied only
src/andtests/, so the unmutated copy failed on the missingtools/make-land.pyand every mutant "died" of that; the service runner counted a mutant that did not compile as killed. A runner that ran the checks on the right tree killed 42 of 44 and showed two missing land checks.[measured: 2026-10-12] - The launcher keeps its choice of themed icons in its own preferences (
icon_theme_idof its launcher preferences). An app cannot read it, so it cannot follow it. The colours and the construction of a themed icon are those ofMonoIconThemeControllerinframeworks/libs/systemui/iconloaderlib.[verified-in-source] Theme.DeviceDefault.DayNightbrings the stock action bar and has noNoActionBarvariant; the bar is dropped withwindowActionBarfalse andwindowNoTitletrue.PackageManager.getApplicationIconreturns the platform's default robot for an app with no icon; testApplicationInfo.iconfirst.- A view with an elevation (the floating bar) draws above siblings added later, so a screen added over the content later still lies under the bar; its end needs the bar's height as padding.
- The history file is in credential-encrypted storage and unreadable from the shell. On a
userdebugimageadb rootreads it, andadb unrootrestores the shell.bmgr backupnow --allonLocalTransportshows what a backup writes; the backup manager and the transport are put back afterwards. adbmay show another phone than the test device: check the serial before any command and useadb -s.- A whitelist for untrusted text must name the ASCII set it accepts.
str.isalpha,str.upperand the regex class\daccept more: a long s counted as the path commandSand Arabic-Indic digits counted as numbers inimport-logos.py, until a review found it (2026-10-12). The same check must count the numbers of every path command, because Android readspathDatawhen the drawable is drawn, not when the package is built[assumption: not run on a device]. - The logo tool refuses a svg with any attribute outside its list. Wikimedia Commons files carry
version,idandpreserveAspectRatio, and 34 of 34 files fetched were refused, so Commons gave no logo; the tool was not relaxed, because reading untrusted svg is the owner's call. A logo is drawn in one colour (fillColorof the drawable), so a logo of several colours would become a solid shape in any case. - A
VectorDrawablethat is faded withsetAlphaarounddraw()stays opaque on a hardware canvas[assumption: the reviewer's recollection of the framework, not run on a device];GlobeViewfades the logo withsaveLayerAlpha. Look at a logo near the rim on the device (MAP-024). - When a plan copies a directory to the build server, compare it with what the server repository holds: the plan's
rsyncofclassifier/would have addedclassifier/data/, whichvendor/oblivios/datamaphas never held.rsyncwith several remote sources fails unless--files-fromis used, and the shell reported exit code 0 for the script that ran it: read the output, not the code. - A mutation runner goes stale when classes are removed or functions move: the app runner of the Atlas branch failed on a deleted
Cluster.javaand 10 entries no longer matched their text. Check every entry'soldtext against the current file before a run (the run itself stops at the first broken entry), and expect the mutations to expose missing assertions: three checks (the count of the type Unknown, the order by shared apps, a tab that does not exist) were added after survivors (129 of 129 killed, 2026-10-13). A mutation that changes every longitude or inserts a no-op is an equivalent mutant and proves nothing. - A tree equal to the reference branch proves the transcription, not the design: the reviewer found real defects in the reference itself (2026-10-13), among them a full-screen state lost after a theme change and a check that passed whatever the length check did (
{"9","x"}was refused for its tab, not its length). A check counts only if the mutation it should catch fails it; 15 of 40 extra mutations the reviewer tried had survived 129 entries. - A shared
Drawablemust not go to two views: each sets its own bounds. Cache the loaded drawable and hand every caller a copy of its constant state (Apps.icon). The app draws the system's own icon (ADR-162); the themed path needed the monochrome layer, which only some apps ship (C13). - A mutation runner is a script that runs when imported; give it a
__main__guard. In the tool shell,pkill -fandps | grep | killmatch the shell's own command line and end the call with code 144: look up the pid and kill it by number. - Code in a plan is safest when a script writes it from the tested commit: the Atlas plan's blocks are the files of the branch, and a replay on the base reproduced the branch tree byte for byte. Colour files, icons and the texture fixture are generated from the design page and regenerate to the same bytes.
- A wrong view type in a constructor (
(ImageView)on theFrameLayoutthat wraps the icon) passes the type-check, every host check and both reviews, and ends the activity on the first start. The view layer has no host test: the first run onlynxis the check (2026-10-13, image2026101203,MapTab.<init>). - A system app cannot be hot-installed with
adb install -ron this image:INSTALL_FAILED_INTERNAL_ERROR: fs-verity not set up for system package update. A fix of a privileged app needs an image. - A frame of 93 ms with 9 ms on the GPU is CPU work:
dumpsys gfxinfo <package>for the frame percentiles, then on auserdebugimageadb rootandsimpleperf record -p <pid> -g --duration 6. 77 percent of the app's CPU was Skia'sSoftwarePathRendererrasterising the land (93 000 points, filled and stroked four times per frame). A complex path drawn per frame is drawn on the CPU; draw it from a texture through an AGSLRuntimeShader(API 33) instead.[measured: MAP-026] settings delete global animator_duration_scaleaftersettings put ... 0leaves the live scale at 0: the fallback of the window manager is its current value. Set1.0explicitly. Count the frames of a tab switch withdumpsys gfxinfo(1 to 2 frames: no motion; 23 frames: the animation ran).- Read an Android screen through
uiautomator dumpfor bounds andscreencapfor the picture, tap from the same dump, and read the state back from a fresh dump (a stale dump once toggled a switch here). CompoundButton.performClickalready callstoggle(): a custom switch that overrides it and toggles again flips twice per tap (on, then off in the same millisecond), so nothing seems to change and the listener fires twice. The event log shows it as two events at one timestamp (MAP-036).- Set the drawn state of a custom control before
setCheckedin its constructor: an override ofsetCheckedthat animates from the field's default makes every control built "on" slide from off. The page is built again after every tap, so every switch that was on moved with the one that was tapped. - A page that is built again at every live snapshot takes the touch away from a finger that is on it. Hold the rebuild while a finger is down and run it on the up event, and leave a page alone when the data it shows did not change. An
adb input tapnever lands in the middle of an update, so it cannot show this: the finger test is separate evidence (MAP-036). adb shell input keyevent BACKon the Map tab with nothing open leaves the app for the launcher; read the focused window before the next tap.- A history file is deleted when the history is switched off, so a double toggle of that switch costs the rows of the day.
- Flags are resources, not a font: 257 lossless WebP files from one pinned tarball (
datamap/app/tools/make-flags.sh, a second run is byte-identical), a 4 by 3 drawable with rounded corners; a card reserves the width of the flag whether or not the set has one. - A page that is added is laid out at the next frame; a touch that comes before it lands on views with no size and is lost. Built again after every change, a page loses about one tap in twenty at a tap rate of 2 per second. Move the controls of a page in place instead, hold a snapshot that is older than the owner's last change, and call
Ds.layoutNowafter any rebuild that stays. - Measure a finger problem with the finger:
adb shell getevent -ltfor the touches,logcat -b eventsfor what the service did,screenrecordfor the animation, then match each tap to its event by time (a tap with no event is the lost one). 24adbtaps found nothing that 45 of the owner's taps found.