# How you investigate this, and why it works The companion to `the-attack.md`. That document explains how the vulnerability works. This one explains why an investigation of it is possible at all, what each piece of evidence proves, and where the method runs out. Read this before running anything. The scanner is twenty minutes of work to operate and the reasoning below is what makes its output mean something. The Active Storage internals cited here were checked against 8.1.0. They are stable in substance across the versions this vulnerability spans, but exact method names and call sites move, so confirm against the version your application was actually running rather than taking a name on faith. ## What the investigation is for The question is not academic. If someone read `config/credentials.yml.enc` or the process environment, you are rotating `secret_key_base` and everything it signs, and that is disruptive enough that you want it to rest on evidence rather than on the fact that a vulnerability existed. So the goal is a defensible answer to three questions, in order: 1. Did anyone stage this attack against us? 2. If so, did any of it actually run? 3. If so, exactly what came back? The durable record is the object store and the Active Storage tables, because those are what survive for years. Application logs are usually retained for weeks, so they cannot bound a search over the exposure window. That does not make them a footnote. Once the sweep has narrowed years down to a handful of objects, logs answer nearly every question the database cannot, and they frequently decide the outcome. There is a section on them below, and it is worth reading before you start rather than after, because log retention is a clock that is already running. ## What the attack leaves behind Three artifacts, in increasing strength. | # | Artifact | What it proves | |---|---|---| | 1 | An unattached blob holding the crafted MAT file | Somebody staged the attack | | 2 | A row in `active_storage_variant_records` for that blob | libvips transformed it, so the read ran | | 3 | The variant image attached to that row | What actually left the building | The third is the unusual one and it is why this vulnerability is more investigable than most file reads. libmatio writes the bytes it read into the variant as pixel values, and Active Storage stores that variant as an ordinary blob. The exfiltrated data is sitting in your own object store, and you can read it back rather than assuming the worst. ## Why the crafted blob survives This is the load-bearing assumption of the whole method, and it deserves checking against your own application rather than being taken on faith. **The attack does not attach the blob to anything.** `Blob#variant` gates only on `variable?`, which reads the declared `content_type`, and the representations controller resolves the blob from `signed_id` and the transformation from `variation_key` independently of each other. So an attacker needs no owning record, and there is usually no reason for them to create one. **Purge paths run through owning records.** `has_one_attached` and `has_many_attached` default to `dependent: :purge_later`, explicit purges hang off models, and cascading deletes of accounts or projects reach attachments through their owners. An unattached blob has no owner, so none of those paths reaches it. **Which leaves scheduled cleanup, and this is the one to check.** Many applications run a job that collects unattached blobs, of the shape the Rails guides suggest: ```ruby ActiveStorage::Blob.unattached.where(created_at: ..2.days.ago).find_each(&:purge_later) ``` That job selects on exactly the criterion this investigation selects on, and purging the blob destroys its variant records too. An application running one has a scan horizon set by that job's retention rather than by the exposure window, and everything earlier is unobserved rather than clean. Search for it before you believe an empty result: ```bash grep -rn "unattached" app lib config db ``` Check `config/recurring.yml`, `config/schedule.rb`, cron, and any rake task the deploy invokes. An application that collects nothing on a schedule still has unattached blobs from years ago, and that is what makes a long window searchable. ## The blind spot **An empty scan is evidence, not proof.** An attacker might be able to delete their own crafted blob along with every artifact derived from it. They do it by attaching it to something and then removing that something. Any route in your application that accepts a blob **by signed id** rather than as a file upload will do, because Active Storage never re-identifies a blob that arrives that way. Attach, then destroy or replace, and `has_one_attached` purges the blob it replaced. The purge takes the evidence of success with it. `ActiveStorage::Blob::Representable` declares `before_destroy { variant_records.destroy_all if ActiveStorage.track_variants }`, and each variant record holds its rendered image through `has_one_attached :image`. Destroying the source blob destroys its variant records and purges the rendered output that held the leaked bytes. Worth stating plainly, because it is easy to misread: the attachment plays no part in the read itself. The attacker creates the blob at the direct uploads endpoint, performs the read against that blob's signed id at the representations endpoint, and only afterwards attaches and removes it, purely to clean up. Two things bound this. **It gains the attacker nothing except cleanup.** An unattached blob is already variable and already addressable by signed id, so attaching is pure overhead unless you know the cleanup trick exists. **It leaves request-level traces.** If the activity falls inside your log retention, you can look for the destroy path being used and settle it for that period. That is worth doing, and it is what turns "we cannot rule this out" into "we ruled it out for the window we can see." ### Finding the routes that accept a signed id This is worth doing while you are in the code anyway, both because it bounds the blind spot and because it is a hardening finding in its own right. A multipart file upload is safe on this axis. Attaching an `ActionDispatch::Http::UploadedFile` builds a new blob through `ActiveStorage::Blob.build_after_unfurling`, and `ActiveStorage::Attached::Changes::CreateOne` calls `identify_without_saving` on it. Marcel re-identifies the bytes, a MAT file is stored as `application/x-matlab-data`, and `variable?` fails. Attaching by signed id finds an existing blob instead of building one, so nothing re-identifies it and the client-declared type stands. The dangerous shape is a controller that does something like `record.image.attach(params[:image_sgid])`, taking a signed id from a parameter. Grep for `attach(` and look at what is being passed. Note that client-side type checks and validations gated on `if: :api_request?` or similar carry no weight here, because the whole exchange is two plain HTTP requests that never run your JavaScript and may not take the validated branch. ## The procedure ### Step 1. Build the candidate set from the database The obvious formulation is to take every unattached blob in the window and check each one: ```ruby ActiveStorage::Blob.where.missing(:attachments).where(created_at: window) ``` **Do not do this.** It loads a full blob row for every unattached blob over years, which on a large application is a great many rows to hold in memory for no benefit. The scanner offers it as `--all-candidates` and it is not the normal mode. Turn it around instead, and apply the variant-record test first, because it is far more selective: ```ruby transformed = ActiveStorage::VariantRecord.where(blob_id: window).pluck(:blob_id, :variation_digest) attached = ActiveStorage::Attachment.where(blob_id: window).distinct.pluck(:blob_id) ActiveStorage::Blob.where(id: transformed.map(&:first).uniq - attached) ``` Both of those are index-only range scans on the stock Active Storage indexes, and only the small difference set is loaded as full rows. That is what the scanner does, and it is why a window covering years comes back with a few hundred candidates rather than millions. The window is a range of blob ids rather than a `created_at` predicate. Blob ids are auto-increment, so `created_at` rises with id to within the width of a concurrent transaction, which is close enough to seek a forensic window and lets both scans stay on the id indexes. Each candidate row carries `key`, the declared `content_type`, `filename`, `byte_size`, `checksum` as a base64 MD5, `created_at` and `metadata`. That gives real timestamps with no decoding, and the checksum lets you group identical objects without downloading them. Unattached is normal and this set is not suspicious by itself. Abandoned upload forms leave unattached blobs behind, as do any endpoints that create a blob and hand back a signed id the client never uses. **Do not filter on `content_type`.** It is tempting, because a declared image whose bytes are not that image is the type confusion itself. But it is an attacker-controlled column that anything could have rewritten afterwards, and the next step is a strictly better test that costs no more. ### Step 2. Confirm by content Read the first 128 bytes of each candidate from the object store and test the two header fields. `the-attack.md` explains why those two, and why nothing legitimate carries both. Read 128 bytes, not the whole object. Some candidates are real images running to hundreds of megabytes, and you have no reason to download one. **Do not test for the HDF5 signature at offset 512.** The crafted files in the original proof of concept used a 512-byte userblock, so this looks like a free third check. HDF5 accepts any power of two from 512 upward, matio reads only the first 128 bytes before delegating, and a container beginning at 1024 or 4096 works identically. A detector anchored to a fixed offset produces false negatives on exactly the files you care about. ### Step 3. Understand what the variant record already told you Step 1 folded this test into the selection, because it is the most selective criterion available. It is worth being explicit about what it bought, since it is what separates a staged attempt from a successful read, and you have it only if `ActiveStorage.track_variants` was enabled. Every generated variant writes three things: a row in `active_storage_variant_records` holding `blob_id` and `variation_digest`, an `active_storage_attachments` row whose `record_type` is `ActiveStorage::VariantRecord` and whose `name` is `image`, and a new blob for the rendered image. So a candidate that reached Step 2 at all was transformed, and the third of those artifacts is what Step 4 goes after. The variant record survives for the same reason the candidate does: its source blob is unattached, so nothing purges the source, and nothing purges the record hanging off it. If you want to sweep staged attempts as well as successful ones, that is `--all-candidates`, which drops this criterion and keeps only the attachment test. It is much slower, it answers a different question, and a crafted blob it finds without a variant record leaked nothing. ### Step 4. Recover what leaked Download the variant image for any confirmed candidate. It is the rendered output, so the target file's bytes are its pixel values. Read them back rather than assuming the worst, and you learn which bytes left, from which file, at the `created_at` on that variant blob. Then read the target path out of the payload's own HDF5 dataset creation property list, which gives you the authoritative `(path, offset, length)` triple rather than strings scraped out of the bytes. The two measurements should reconcile: bytes requested according to the property lists, pixels returned according to the variants. Agreement across two artifacts derived by different means is the strongest integrity check available here. ## What the logs are for The database tells you what happened. The logs tell you who did it, whether they cleaned up afterwards, and whether anyone else tried. Do not treat this as an optional final flourish; on a real investigation it is where most of the conclusions come from. **Short retention matters less than it sounds.** It is true that logs cannot cover a multi-year window. But investigations of this kind are almost always prompted by a disclosure, and activity triggered by a disclosure is recent by construction, so the period you most want to examine is usually the period still in retention. Check the dates before assuming otherwise. **Pull the logs early.** Retention is a clock that started before you did. If the sweep is going to run for hours, start the log extraction alongside it rather than after. ### The join is the checksum Log lines rarely let you filter on a blob checksum, and that does not matter. The sweep has already given you the exact checksums of the crafted objects, and there are usually only a handful of distinct ones, so a plain line filter on each locates the upload requests directly. `byte_size` corroborates independently, because crafted files of a given shape are all the same size. This is why the order matters. Narrowing with the database first turns an impossible log search into a search for a dozen literal strings. ### What each query answers **Who uploaded it.** The account, the authenticated user, the client address, and the endpoint. This is the only source for any of it, since an unattached blob has no owning record. **Whether it was one actor or several.** Requests that were rejected, by CSRF protection, rate limiting, or authentication, never create a blob, so the sweep is blind to them entirely. Someone who tried and failed exists only in the logs. An investigation that skips this can report one actor where there were two. **Whether they cleaned up.** Look for the destroy and update paths on whatever routes accept a blob by signed id. This is the blind spot described above, and finding those requests absent is what lets you say the count is not an undercount, for the window you can see. It is the highest-value log query in the whole investigation and it has nothing to do with attribution. **Whether the counts reconcile.** Upload requests, then requests to the representations endpoint, then variant records, should form a descending series. The gaps are meaningful rather than alarming: staged files that were never rendered leaked nothing, which is the entire basis for using a variant record as the success test. A series that does not descend means you have misunderstood something. **What the tooling looks like.** User agent is worth reading rather than skipping. A bare or scripted agent on the uploads, alongside an ordinary browser agent on the same account's normal navigation, is someone driving an exploit from a terminal while browsing the product in a window. Named tooling in the agent string sometimes identifies the party outright. **Sometimes, what leaked.** If the application logs image dimensions during processing, and the payload used the dimension-encoding technique rather than pixel encoding, those logged dimensions are the exfiltrated bytes. This is worth checking whenever a payload carries `MATLAB_empty`, and it is the only way to recover a read whose output was never stored. ### What logs cannot do They cannot bound the search, they expire, and they say nothing about the period before retention. A finding of "no attribution possible, the activity predates retention" is a legitimate and complete answer. Do not read benignity into an absence of log evidence. ## Two other signals worth checking None of these is necessary, and each has caught something the main procedure would not. **Blobs flagged as having failed a representation.** If your application records failed representation attempts, a candidate carrying that flag had a representation attempted that did not finish. The scanner reports `metadata[:preview_failed]` for this reason. It is a weaker signal than a variant record, because it says an attempt was made rather than that it succeeded. **Analyzer metadata that does not match a real image.** Attaching a blob triggers analysis, and `ActiveStorage::Analyzer::ImageAnalyzer.accept?` is `blob.image?`, which is `content_type.start_with?("image")`. So a spoofed blob is accepted for analysis on the strength of its declared type, exactly as it is accepted for variant processing. Analysis records only width and height, so it is not an exfiltration channel by itself. But a crafted blob that was ever attached carries dimensions in `metadata` matching its MAT dataset rather than any real image, which is one way to find a blob whose variant records are gone. Note the precondition: this only applies to blobs that were attached, and the attack does not attach them, so it is a signal about the cleanup path rather than about the read. ## Limits Every one of these belongs in the write-up, including on a clean result. **Object store lifecycle rules are invisible from the application repository.** If the bucket carries an expiration or transition rule, the retention this whole method depends on does not hold, and objects may be gone regardless of what the database says. Somebody who owns the storage has to confirm this. It is the limit most often missed, because nothing in the code can tell you about it. **Scheduled cleanup of unattached blobs**, covered above, sets the real scan horizon when it exists. **The attach-then-purge blind spot**, covered above. Logs close it for the period they cover. **Variant tracking must have been on for the whole window.** It follows `load_defaults` and became the default at Rails 6.1. Confirm when the application crossed that before relying on the absence of variant records for earlier dates. **Log retention bounds attribution.** Blob rows carry no owner, and these blobs are unattached by construction, so the database cannot tell you who uploaded them. Beyond retention you can say what was read and not by whom. **The detector tests the attack as it has been described.** Two header fields is what the known chain has to get wrong together. That is a claim about traced code paths, not a theorem, and a variant nobody has published would not be caught.