ObliviOS Developer portal Project site

Home / Engineering notes / Data map

Data Map engineering notes

Lessons and tool behaviour found while building the Live Data Map.

Lessons and tool behaviour found while building the Live Data Map (ADR-157, ADR-158): the collector in system_server, the classifier file and DataMapApp. The design is 2026-10-11-live-data-map-design.md; the patches, their baselines, the landing and the module build are recorded in datamap/SOURCE.md of the oblivios-platform repository. Claims are labelled [verified-in-source] (the path is named) or [measured] (the run is named).

1The netd callback thread must not block

NetdEventListenerService.onDnsEvent and onConnectEvent are synchronized and call each registered callback with that monitor held, on a binder thread of the resolver; their own comments say they "must not block or perform long-running operations" [verified-in-source: frameworks/base services/core/java/com/android/server/connectivity/NetdEventListenerService.java]. The first version of the collector asked the activity manager (getUidState takes mProcLock, [verified-in-source: .../am/ActivityManagerService.java]) and the package manager from that thread, so a slow lock would have stalled every consumer of the events. The collector now validates the name and hands the event to the service thread through a bounded counter (PendingLimit, 4096); an event that finds no place is dropped and counted.

Unrelated to the thread, but easy to misread: the callback runs inside a transaction from netd, which is root, so Binder.getCallingUid() there is root and the package filter lets every package through [verified-in-source: .../pm/AppsFilterBase.java]. Attribution was right by accident of that uid, not by design.

2Order of events at user stop

UserController.finishUserStopped calls onUserStopped and then, synchronously, locks the user's credential-encrypted storage [verified-in-source: .../am/UserController.java]. A write only posted from onUserStopped runs after the lock and is lost. The final write belongs in onUserStopping, and a receiver for ACTION_SHUTDOWN writes at shutdown.

3What a type-check cannot see: the first module build

A type-check of the new classes against the jars of an earlier build passed, and the first m of the modules still failed twice:

  1. fuzzer_bindings_test refuses a new service that is not listed in system/sepolicy/build/soong/service_fuzzer_bindings.go. A Java service has no automatic fuzzer (b/287102710); the exception is EXCEPTION_NO_FUZZER.
  2. A <permission> declared in core/res/AndroidManifest.xml becomes a public Manifest.permission constant, and check_current_api refuses a public API that core/api/current.txt does not list. A permission for one app of this system needs @hide in the comment above it.

m -k shows every failure of a run; without it a run stops at the first. Soong accepted the new Android.bp files (prebuilt_etc with sub_dir and filename, android_app with AIDL) without change [measured: module builds of 2026-10-12].

4SELinux

  • system_app and traceur_app are app domains with an allow for every service type except a fixed list [verified-in-source: system/sepolicy private/system_app.te, private/traceur_app.te]. A new service type that only one app domain may find needs -datamap_service in both lists, or the neverallow for the other app domains does not compile.
  • enforceCallingPermission is satisfied by app id 1000 (the activity manager grants every permission to it), so the permission does not separate the Data Map app from the apps of the system uid; SELinux is the only gate.
  • A new system service needs entries in service.te, service_contexts and, for a privileged app with its own domain, seapp_contexts (the pattern of logviewer_app).

5Bounds of the reply

A Binder transaction is about 1 MB. A snapshot of the Data Map is bounded in the service (2000 entries, 300 links) because links are derived for every pair of apps that share a tracker and grow with the square of the apps. The history is read once per change of the file and kept for the open app only. A snapshot over the limit would arrive as a failed transaction exactly when the data is richest.

6Server places: bounds the review found

  • A decoder bound that was sized for one format is not the bound of the next. The line limit of the history decoder (600) was sized for the ten-field row; a place of two names of 100 characters made a row of 634, the decoder refused the file, and the next write started from nothing. The limit is now derived from the maxima (1024) and the encoder keeps to it: a place that would pass it is left out, a row that is too long without one is not written.
  • A reply is bounded by bytes, not only by rows: seven fields with names of up to 255 characters take up to 2 KB an entry. DataMapStore.within keeps the newest rows that fit 600 KB, counted as the parcel holds them.
  • A flag added to a file that an earlier build reads must not be a new key of that file when the reader treats an unknown key as "use the defaults": a rollback would have switched collecting on. The choice to keep server places is a file of its own.
  • Unsigned values in Java: AS numbers of 2**31 or more are negative ints, and IPv6 keys with the top bit set sort first when compared as signed. A fixture that holds neither passes with the wrong compare: the fixture now has a private AS number of 4200000000 and a range at a000::. A mutation that cannot be killed is an equivalent mutant: the 48-bit anycast prefixes never have the top bit set, so the signed compare of member64 is not a mutation to keep.
  • Character.digit accepts the digits of other scripts (a fullwidth 1 was an IPv6 digit); a parser of untrusted text names its ASCII set. A zone id belongs to an IPv6 address only; 8.8.8.8%x is not an address.
  • Python str.isalpha takes a letter outside ASCII, which does not fit the 16 bits of a country code: pack_cc crashed the build tool on such a row.
  • A test that compares a model with itself proves nothing about the wording: the check asserted that Frankfurt and Berlin are one place and never read what the card says about them. A card names the lead only, so a place that joins cities must not be named after one of them; a server place is never joined.
  • A sheet is not the globe: the rule "a server place is joined to nothing" held for the lines (PlaceLinks.pairs) and not for the list under the sheet (PlaceLinks.joined). Put a rule in the function every caller goes through.
  • A record that says a file is archived must name the path: the inputs of the first ipmap.bin were said to be archived and were nowhere on the build server, so its pins could only be met by luck (the latest release had not changed that day). The inputs are now kept in datamap-inputs/ on the build server and checked against PINS.sha256.
  • Landing a series on the build server: compare the blob of every touched file with the disposable tree after git am (git rev-parse HEAD:<path> against git hash-object); the AIDL files are not in the tree that materialize-core.sh writes, so compare them from the patches. A remote grep -c '[s]oong_ui' counts its own command line when the same text is elsewhere in the command (an echo soong_ui: did): look at ps output, not at a count.
  • Launching a build over ssh: A && B && nohup script & disown puts the whole chain in the background and keeps the ssh call open on its output. Start the script as its own command and read the log from a second call.
  • Merging a feature line into main that has moved on (the ADR-163 squash): use a temporary worktree of main and git merge --no-ff, then git diff <branch> main -- datamap must be empty.
  • A replay script of edits goes stale the moment the code is changed by hand; the two edit lists of the branch were removed, the patch and the sources are the record.

7Switch defects found on the device (2026-10-13 and 2026-10-14)

  • Two Switches with one view id restored each other's value on a theme change; the interface review found it as a Critical defect and each Switch has its own id now.
  • A tap on a switch toggled twice (MAP-036): a constructor order made the Settings page set a switch before its listener existed. A page built again under a finger then lost a tap now and then, also when it was built again before its layout (captured with the touch log). Settings therefore moves its own controls in place instead of building the page again.

8The classifier data

  • Sources and licences: datamap/classifier/SOURCES.md in oblivios-platform. The tracker lists of Disconnect, DuckDuckGo and Ghostery are CC-BY-NC-SA-4.0 and cannot ship. The AdGuard companies database is CC-BY-SA-4.0 (built on whotracks.me, CC-BY-4.0); the classifier file therefore carries CC-BY-SA-4.0 and a NOTICE.
  • The database has companies, domains and categories but no country. Wikidata (CC0) gives a headquarters for about 15 percent of its companies, found by the official website; Wikidata has no item with a headquarters for the other, mostly small advertising companies [measured: tools/fetch-wikidata-bases.py, 2026-10-12]. Their services are listed with a category and no owner. The search API of Wikidata answers HTTP 429 after a few hundred requests; SPARQL does not.
  • The device parser caught what the builder had not: two records of the database that point to one Wikidata item gave a duplicate source, and a Wikidata item with no English label returns its code (Q123) as label, which had become a legal name or a city in 22 owners. The importer merges the first and refuses the second.
  • An owner id is [a-z0-9_]{1,24} (no hyphen).

9Process lessons

  • A research agent is not a source. Every row of the hand-checked layer was read against its page by the author, and rows were dropped: a partnership whose 10-year term had ended, a category the page only implied, a domain that serves unrelated content as a whole (unity3d.com).
  • Relations are shown as current. Only owns and merged_with stay true once done; partner_of and buys_ad_space_from need a source that shows the arrangement is current.
  • A mutation count proves nothing until the unmutated tree has passed first and every mutant compiles. The app runner copied only src/ and tests/, so the unmutated copy failed on the missing tools/make-land.py and every mutant "died" of that; the service runner counted a mutant that did not compile as killed. A runner that ran the checks on the right tree killed 42 of 44 and showed two missing land checks. [measured: 2026-10-12]
  • The launcher keeps its choice of themed icons in its own preferences (icon_theme_id of its launcher preferences). An app cannot read it, so it cannot follow it. The colours and the construction of a themed icon are those of MonoIconThemeController in frameworks/libs/systemui/iconloaderlib. [verified-in-source]
  • Theme.DeviceDefault.DayNight brings the stock action bar and has no NoActionBar variant; the bar is dropped with windowActionBar false and windowNoTitle true. PackageManager.getApplicationIcon returns the platform's default robot for an app with no icon; test ApplicationInfo.icon first.
  • A view with an elevation (the floating bar) draws above siblings added later, so a screen added over the content later still lies under the bar; its end needs the bar's height as padding.
  • The history file is in credential-encrypted storage and unreadable from the shell. On a userdebug image adb root reads it, and adb unroot restores the shell. bmgr backupnow --all on LocalTransport shows what a backup writes; the backup manager and the transport are put back afterwards.
  • adb may show another phone than the test device: check the serial before any command and use adb -s.
  • A whitelist for untrusted text must name the ASCII set it accepts. str.isalpha, str.upper and the regex class \d accept more: a long s counted as the path command S and Arabic-Indic digits counted as numbers in import-logos.py, until a review found it (2026-10-12). The same check must count the numbers of every path command, because Android reads pathData when the drawable is drawn, not when the package is built [assumption: not run on a device].
  • The logo tool refuses a svg with any attribute outside its list. Wikimedia Commons files carry version, id and preserveAspectRatio, and 34 of 34 files fetched were refused, so Commons gave no logo; the tool was not relaxed, because reading untrusted svg is the owner's call. A logo is drawn in one colour (fillColor of the drawable), so a logo of several colours would become a solid shape in any case.
  • A VectorDrawable that is faded with setAlpha around draw() stays opaque on a hardware canvas [assumption: the reviewer's recollection of the framework, not run on a device]; GlobeView fades the logo with saveLayerAlpha. Look at a logo near the rim on the device (MAP-024).
  • When a plan copies a directory to the build server, compare it with what the server repository holds: the plan's rsync of classifier/ would have added classifier/data/, which vendor/oblivios/datamap has never held. rsync with several remote sources fails unless --files-from is used, and the shell reported exit code 0 for the script that ran it: read the output, not the code.
  • A mutation runner goes stale when classes are removed or functions move: the app runner of the Atlas branch failed on a deleted Cluster.java and 10 entries no longer matched their text. Check every entry's old text against the current file before a run (the run itself stops at the first broken entry), and expect the mutations to expose missing assertions: three checks (the count of the type Unknown, the order by shared apps, a tab that does not exist) were added after survivors (129 of 129 killed, 2026-10-13). A mutation that changes every longitude or inserts a no-op is an equivalent mutant and proves nothing.
  • A tree equal to the reference branch proves the transcription, not the design: the reviewer found real defects in the reference itself (2026-10-13), among them a full-screen state lost after a theme change and a check that passed whatever the length check did ({"9","x"} was refused for its tab, not its length). A check counts only if the mutation it should catch fails it; 15 of 40 extra mutations the reviewer tried had survived 129 entries.
  • A shared Drawable must not go to two views: each sets its own bounds. Cache the loaded drawable and hand every caller a copy of its constant state (Apps.icon). The app draws the system's own icon (ADR-162); the themed path needed the monochrome layer, which only some apps ship (C13).
  • A mutation runner is a script that runs when imported; give it a __main__ guard. In the tool shell, pkill -f and ps | grep | kill match the shell's own command line and end the call with code 144: look up the pid and kill it by number.
  • Code in a plan is safest when a script writes it from the tested commit: the Atlas plan's blocks are the files of the branch, and a replay on the base reproduced the branch tree byte for byte. Colour files, icons and the texture fixture are generated from the design page and regenerate to the same bytes.
  • A wrong view type in a constructor ((ImageView) on the FrameLayout that wraps the icon) passes the type-check, every host check and both reviews, and ends the activity on the first start. The view layer has no host test: the first run on lynx is the check (2026-10-13, image 2026101203, MapTab.<init>).
  • A system app cannot be hot-installed with adb install -r on this image: INSTALL_FAILED_INTERNAL_ERROR: fs-verity not set up for system package update. A fix of a privileged app needs an image.
  • A frame of 93 ms with 9 ms on the GPU is CPU work: dumpsys gfxinfo <package> for the frame percentiles, then on a userdebug image adb root and simpleperf record -p <pid> -g --duration 6. 77 percent of the app's CPU was Skia's SoftwarePathRenderer rasterising the land (93 000 points, filled and stroked four times per frame). A complex path drawn per frame is drawn on the CPU; draw it from a texture through an AGSL RuntimeShader (API 33) instead. [measured: MAP-026]
  • settings delete global animator_duration_scale after settings put ... 0 leaves the live scale at 0: the fallback of the window manager is its current value. Set 1.0 explicitly. Count the frames of a tab switch with dumpsys gfxinfo (1 to 2 frames: no motion; 23 frames: the animation ran).
  • Read an Android screen through uiautomator dump for bounds and screencap for the picture, tap from the same dump, and read the state back from a fresh dump (a stale dump once toggled a switch here).
  • CompoundButton.performClick already calls toggle(): a custom switch that overrides it and toggles again flips twice per tap (on, then off in the same millisecond), so nothing seems to change and the listener fires twice. The event log shows it as two events at one timestamp (MAP-036).
  • Set the drawn state of a custom control before setChecked in its constructor: an override of setChecked that animates from the field's default makes every control built "on" slide from off. The page is built again after every tap, so every switch that was on moved with the one that was tapped.
  • A page that is built again at every live snapshot takes the touch away from a finger that is on it. Hold the rebuild while a finger is down and run it on the up event, and leave a page alone when the data it shows did not change. An adb input tap never lands in the middle of an update, so it cannot show this: the finger test is separate evidence (MAP-036).
  • adb shell input keyevent BACK on the Map tab with nothing open leaves the app for the launcher; read the focused window before the next tap.
  • A history file is deleted when the history is switched off, so a double toggle of that switch costs the rows of the day.
  • Flags are resources, not a font: 257 lossless WebP files from one pinned tarball (datamap/app/tools/make-flags.sh, a second run is byte-identical), a 4 by 3 drawable with rounded corners; a card reserves the width of the flag whether or not the set has one.
  • A page that is added is laid out at the next frame; a touch that comes before it lands on views with no size and is lost. Built again after every change, a page loses about one tap in twenty at a tap rate of 2 per second. Move the controls of a page in place instead, hold a snapshot that is older than the owner's last change, and call Ds.layoutNow after any rebuild that stays.
  • Measure a finger problem with the finger: adb shell getevent -lt for the touches, logcat -b events for what the service did, screenrecord for the animation, then match each tap to its event by time (a tap with no event is the lost one). 24 adb taps found nothing that 45 of the owner's taps found.

Read from docs/architecture/datamap-engineering-notes.md.