Multimodal Learning | Chang Nie

MID: A Self-Supervised Multimodal Iterative Denoising Framework

Thu, 30 Apr 2026 00:00:00 +0000

The clean version often does not exist

Good data is the precondition for everything downstream, and acquisition noise is nearly universal: sensor limitations, transmission errors, environmental interference, and the stochastic nature of physical measurement itself. The usual response is to denoise first.

Two practical facts make that harder than it sounds, and both shaped this paper.

The first is that the noise that actually matters is rarely Gaussian. Poisson noise in low-light imaging, compression artifacts, and what we would call structured noise, such as a set of line segments that do not correspond to any real structure in the scene, all violate the assumptions that classical filters are built on. The second is more fundamental: for many of the domains we care about, the clean target simply does not exist. In clinical imaging and remote sensing there is frequently no noise-free reference to compare against, and no way to acquire one. The MRI experiments in the paper are scored with proxy measures precisely because clean ground truth is unavailable for those datasets.

So the training signal has to come from somewhere other than a paired clean sample. That constraint is the whole problem.

Why the existing answers are conditional

Classical model-based denoising, from filtering to transform-domain thresholding to non-local means, is tailored to specific noise models and data types. Its effectiveness falls away as soon as the noise statistics deviate from the assumption it was built on.

Supervised deep denoising sidesteps the modelling assumption but needs large paired datasets of clean and noisy samples, which are costly or infeasible to collect in exactly the domains where denoising matters most.

Self-supervised blind-spot methods were the natural answer, and the criticism we make of them is specific rather than general. They avoid pairing, and they carry hidden modality assumptions. The tricks that make a blind-spot network work, such as masking pixels and predicting them from neighbours, rely on spatial correlation in images. Point clouds are unordered, and amino acid sequences are discrete and unstructured. A method that assumes nothing about the data can travel to those settings. One that quietly assumes spatial smoothness cannot, and the observable symptoms are oversmoothing and loss of fine detail.

Domain-specific specialists have their own conditionalities. Noise2Noise can oversmooth fine anatomical structures. Patch2Self exploits redundancy across volumes but requires training a separate model per volume, which risks spatial inconsistency between them. DDM2 is sensitive to the initial noise level and to modality-specific tuning. In surface electromyography, high-pass filtering removes the ECG interference and removes important low-frequency components of the signal along with it, because the two overlap in frequency; template subtraction assumes a rigid distributional form; and neural approaches can be unstable to train. For protein sequence curation, the standard greedy selection on average Hamming distance rests on a single criterion that does not capture sequence quality more broadly.

We summarize the positioning as a capability matrix rather than a benchmark table. The question is which methods are self-supervised and work on images, signals and point sets at once. Most of the field manages two of the three.

What makes it hard

The inverse of a nonlinear corruption has no usable closed form. Reversing Poisson noise, compression, or a set-structured nuisance directly is difficult, and our own ablation shows what happens if you try: a single-pass network leaves the data essentially unchanged. Networks struggle to learn a large jump from noisy to clean in one step. That result is not an implementation failure. It is the reason the method is shaped the way it is.

You also do not know how corrupted the input is. Nothing about a noisy sample announces its position on a corruption trajectory, so there is no way to decide how much to remove or how many steps to take.

Self-supervision has to be manufactured rather than found. Without pairs, the only available labels are ones you create yourself, and they have to serve two purposes: how far along the corruption the input sits, and what the next increment looks like.

And sometimes the noise is not a value at all. An incorrect correspondence, an invalid line segment, or a redundant sequence is not a magnitude added to anything. It is a decision about an element. Before any of that can be subtracted, the nuisance has to be recast as a per-element property, which is why those tasks use a binary classification objective rather than a regression on residuals.

Unordered and discrete data break the usual priors outright. A point cloud has no ordering and a multiple sequence alignment is a set of discrete strings, so weight sharing over a spatial grid and convolution are simply unavailable.

There is also a tradeoff baked into the formulation. The first-order approximation underneath our method only holds if the increments are small. Small increments buy linearity, and then cost you a long inference loop.

Restore by adding noise

The idea we started from is counterintuitive enough to state plainly. To learn how to remove noise, you first add more of it.

A noisy observation is not a degraded version of a clean sample sitting off to one side. It is one state on a corruption trajectory, and you can continue that trajectory yourself. Take the noisy input as the starting point, apply the corruption process again and again, and you now have a labelled ladder: at every rung you know exactly how much noise separates it from the rung below, because you put it there.

The move that makes this usable is a local linearization. Writing the corruption as a general operator and expanding to first order,

$$ s_t \approx s_{t-1} + \Delta s_{t-1}. $$

The step from one rung to the next is treated as additive, which means its inverse is a subtraction. The residual the noise network is asked to predict is nothing more than the difference between two consecutive states that we generated. This is an approximation, and we say so: it implicitly assumes the Jacobian is close to identity, or that its effect is absorbed into the learned mapping.

Everything else follows from that. The reverse step becomes tractable, and the training signal is free.

Two networks, two different questions

The framework asks two separate questions, and keeping them separate is what makes one recipe portable across domains.

The first is: where am I? A step prediction network regresses the input’s position on the trajectory, with the step normalized and a mean-squared objective. This is necessary before any amount of subtraction can be chosen, and it is where the method differs most from diffusion. Diffusion trains against a noise prediction derived from a fixed forward schedule. Here the step is estimated from the data itself, without assuming a fixed forward process.

The second question is: what should be removed? The noise prediction network takes the current state and the estimated step and predicts the increment. The two networks are trained jointly, and the decoupling is deliberate: one decides how far to walk back, the other decides what to take off at each step.

At inference the loop is a few lines. Estimate the current step, then subtract predicted increments until the step count reaches zero. A lightly corrupted input takes fewer steps than a heavily corrupted one, so the iteration count adapts to the data without a schedule.

The reason for iterating at all, rather than removing everything at once, is that many small corrections preserve detail better than one large one. Oversmoothing is what happens when a single pass has to guess the whole distance.

View full-size image ↗

A step predictor and a noise predictor are trained jointly and used in sequence at inference.

View full-size image ↗

The reverse self-supervised restoration loop walks a corrupted input back down the trajectory one increment at a time.

Denoising is a way of framing a problem

The most interesting part of this work, for us, is not the image results. It is what happens when the same formulation is pointed at problems that nobody would call denoising.

Image correspondences become a four-dimensional point cloud, where a match is a point with two image coordinates in each view. Wrong matches are points that do not belong to the structure, and the network learns the broader features of incorrect correspondences rather than relying on a match score alone. Line segments used for vanishing point estimation become an unordered point cloud, and the model is trained by iteratively adding random segments as noise. Multiple sequence alignments are handled by treating redundant sequences with low average Hamming distance as noise. In that last case, the noise consists of real amino acid sequences, which is a fair demonstration that the framework is evaluating data quality rather than a synthetic artifact.

What this shows is that denoising is a task framing rather than a data type. Once the nuisance is expressed as something removable, outlier rejection in geometric estimation, correspondence pruning, line filtering and sequence subsampling turn out to be the same problem wearing different clothes.

Two design details follow from the same insight. Multiplicative noise can be made additive in the log domain, which the paper notes and uses. And for point-set tasks the noise objective is a per-element binary cross-entropy, not an additional penalty term on top of a regression. The network is not asked how much to subtract from a wrong correspondence. It is asked whether that element should be there at all.

View full-size image ↗

The same formulation that removes image noise also identifies incorrect matches.

How it is built

For images and medical volumes, a CNN backbone handles the step predictor with fully connected layers on top, and a CNN encoder-decoder predicts the increments. For point sets and one-dimensional signals, including correspondences, line segments, amino acid sequences and surface EMG, both networks use fully connected layers with a Transformer encoder-decoder. The reason is the one given above: convolutions assume a spatial ordering that unordered sets do not have, so the architecture changes with the data rather than the other way around.

None of the domains get a bespoke formulation. That was the point of building it this way. One method, applied to image speckle, MRI noise, muscle-signal contamination, geometric outliers, wrong correspondences, invalid line segments, and redundant protein sequences, with only the backbone changing.

View full-size image ↗

The nuisance here consists of real sequences rather than synthetic artifacts.

What we take from this

Supervision can be manufactured by continuing a corruption you can simulate, rather than by obtaining the clean endpoint. That single substitution is what makes the framework applicable where paired data does not exist.

Local linearization plus iteration converts an intractable global inverse into a sequence of tractable local ones, and the ablation is the evidence. When the single-pass model fails, the right response is not a larger model. It is decomposition.

Separating where am I from what should be removed is what makes one framework portable. That decoupling, more than any architectural choice, is the reusable part.

Self-supervision strategies carry hidden modality assumptions, and it is worth auditing them. A formulation that asserts nothing about the structure of the data will travel further than one that quietly requires smoothness.

And iteration is a detail-preservation mechanism, not just an optimization convenience.

Where it stops

Extreme noise is the stated failure mode. When the signal structure is almost entirely obscured, iterative subtraction has too little partial structure to work from, and generative priors are the direction we point to for future work.

The second cost is time. Iterating is inherently slower than a single-pass denoiser, which is a bottleneck for real-time use. Knowledge distillation and more efficient sampling are the remedies we suggest.

Two things are visible in the method that we did not fully characterize. The number of corruption steps and the magnitude of each increment are design choices rather than derived quantities, and we did not study the sensitivity to them. Errors in the estimated step also compound across the subtractions that follow, and that propagation deserves more attention than we gave it.

What kind of model this is

It is worth being explicit about one boundary, because it comes up in every conversation about this work. MID is not a generative model. Diffusion models minimize a noise prediction loss derived from a fixed forward schedule and can sample new data from the learned distribution. MID is an iterative, self-supervised noise subtractor that learns the structure of the noise present in its target domain and removes it. The output is a cleaner version of the input, not a new sample.

That distinction is what keeps the framework honest about what it can promise. It removes what it was trained to recognize. Anything else is a different problem.

3D Object Detection and Tracking Based on Lidar-Camera Fusion and IMM-UKF Algorithm Towards Highway Driving

Tue, 04 Apr 2023 00:00:00 +0000

The design constraint behind this work is not accuracy. It is latency.

Most fusion-based 3D detection algorithms run below 15 Hz on a workstation, and a workstation is not what drives a car. An onboard computing centre is slower still, and it is not dedicated to perception: planning, control and the rest of the stack have to run concurrently, which takes a further bite out of the same budget.

Now put a number on what 15 Hz costs. At highway speed the vehicle covers about 1.94 metres between two consecutive frames. For that distance the perception system effectively does not see anything that has just happened, and we call that being time blind. The consequences are not abstract. A system that reacts to where a car was two metres ago is reacting to a situation that no longer exists.

Thirty hertz substantially reduces the blind distance and leaves room for the other modules to run. That target, on affordable hardware rather than a development machine, is what shaped every choice in this paper. It also means we made a deliberate trade: LiDAR-only methods achieve better detection accuracy than fusion methods do, and we did not set out to beat them on that axis. We set out to be competitive while running at a frequency they cannot reach.

What the fusion families cost

There are three ways to combine an image with a point cloud, and each has a price.

Detecting in 2D and lifting the boxes into the point cloud, the frustum approach, makes the whole system depend on the 2D detector. An error in the image branch propagates into 3D with no way to recover it.

Fusing at the feature or proposal level and training end to end produces strong results, and it loses original information in the process of fusing at the region level. These methods also tend to be complex and to run slowly, which is the specific problem we were trying to solve.

The third family projects the point cloud into an image feature map and augments each point with what it finds there. The architecture stays close to a LiDAR-only detector, which is what we wanted, and the pillar encoder keeps the runtime low. We chose this family for exactly that reason.

Within LiDAR-only detection the lineage matters too. Voxel-based 3D convolution was accurate and slow. Sparse convolution improved efficiency without making it fast enough. Pillars, which reduce the problem to 2D convolutions, hit the best tradeoff between accuracy and speed, and we built on that.

For tracking, the choice of a classical filter over a learned one was deliberate rather than conservative. Learning-based trackers perform poorly in real time, and neural data association is time-consuming, so the Hungarian algorithm stays. Batch tracking is ruled out for a different reason: it uses all data, and a driving system only ever has history. 2D tracking is ruled out because it cannot tell you an object’s position and size in 3D.

The closest filter-based alternative is a well-known 3D tracker that is simple and efficient and uses a single constant-velocity model. A constant-velocity model has no way to represent a car that is turning, so its state estimate degrades in exactly the situations where prediction matters most. A monocular tracker uses an extended Kalman filter under one camera. An urban JPDA-IMM-UKF system exists and uses traditional clustering for detection, which produces fragmented trajectories and hurts tracking performance downstream.

IMM itself is standard in aviation and nearly absent in vehicles, and the reason is instructive. Aircraft maneuver drastically: large speed changes, fast heading changes, so a switching model earns its complexity. Vehicle motion is comparatively mild. And with little prior work in the automotive setting, the parameters that IMM needs are hard to set.

What makes it hard

Fusing at full image resolution, per point, at thirty hertz, is a budget problem before it is an accuracy problem. The number of image channels you attach to each point is the knob you turn, and we tuned it as a tradeoff rather than maximizing it.

Data-layer fusion is also fragile in a specific and quiet way. Because each point is matched to an image location, a calibration error puts the sampled feature at the wrong pixel for every point, and the resulting input looks entirely plausible. There is no error signal. The system does not know it is being fed garbage.

Choosing a motion model is a lose-lose if you must choose one. Constant velocity is simple and strong for linear motion. Constant turn rate and velocity handles rotation. Either alone is wrong in the other regime. Switching between them is the answer, and switching introduces a set of prior parameters: the state switching matrix, the state covariance, the process and measurement noise covariances. These are conventionally set from experience, and empirical values are not guaranteed to apply to a different scenario. That is a real methodological problem, not a tuning nuisance.

There is a nonlinearity problem too. Constant-velocity transitions are linear in the state. Constant-turn transitions involve position expressed through velocity and yaw rate multiplied by sine and cosine of the heading, which is not linear, so the filter needs sigma points rather than a plain Kalman update.

Tracking also has to survive gaps and resist noise at the same time. Occlusion causes detections to drop, and being too permissive about starting new tracks admits detector false positives. The two failure modes pull in opposite directions.

And the physics caps detection regardless of architecture. In one of our examples, three very distant cars return only three to five LiDAR points each. Too few points and the detection simply does not fire.

Attach a neighbourhood, not a pixel

The first half of the idea addresses the fragility of data-layer fusion. Instead of sampling a single projected pixel for each point, we sample a 5×5 window of image features around the projection and attach the whole neighbourhood to the point.

The consequence is that the network learns the matching relationship between the two modalities rather than trusting the calibration exactly. A point whose projection lands a pixel or two off still finds relevant appearance features in its window. This was the single most useful conceptual move in the detection half, and it is what makes the difference between fusion that survives calibration error and fusion that quietly degrades.

The features come from a full-resolution extractor, which matters for a related reason: the point has to map to a precise image location for the window to mean anything, and a downsampled feature map destroys that precision. The encoder is VGG-based with two modifications. We reduced the channel count to keep optimization manageable, and we cut the network off in the middle so that the feature map does not shrink too far, which would make it useless for generating fusion data. A top-down decoder upsamples and merges with same-resolution encoder maps, with 1×1 convolutions controlling the output dimension.

View full-size image ↗

A local window learns the correspondence between the two modalities instead of assuming the calibration is exact.

Collapsing the height axis to get the frame rate

The augmented points then go through a pillar encoder. Points are binned onto a bird’s-eye-view grid with cell size and a cap on how many points a cell keeps, with random subsampling beyond the cap. The height axis is collapsed to a single cell, which is the trick that turns 3D convolution into 2D convolution and is where the frame rate comes from.

Each point carries its geometry and appearance plus five offset features that describe where it sits relative to its cell’s mean and its voxel centre. A PointNet layer expands these, max-pooling reduces each cell to one representative, and the cells are scattered back into a pseudo-image. A 2D convolutional backbone and a region proposal head then predict oriented 3D boxes.

The cost of the pillar representation is that height structure inside a cell is discarded and the detection range is bounded by the camera’s field of view, since fusion needs pixels. We accept both as the price of the frequency target.

One loss term deserves a note because it looks redundant and is not. The yaw residual is computed as the sine of the difference between predicted and anchor angle, and sine is symmetric. A car facing forward and a car facing backward produce the same residual. Regression alone cannot break the tie, which is why a separate direction classification term exists in the loss.

View full-size image ↗

The fusion-augmented point cloud is encoded as a bird’s-eye-view pseudo-image.

Two motion models, and a search for the priors

The tracker predicts every existing trajectory, associates predictions to detections by the Hungarian algorithm on 3D overlap, updates matched tracks, and manages births and deaths for everything unmatched.

The interesting part is the update. Two models run in parallel, one with a constant-velocity state and one with a constant turn rate and velocity state. Mixing probabilities are computed from the previous model weights and the switching matrix, mixed estimates are formed for each model, each model’s sigma-point filter predicts forward, and the two are combined into the IMM prediction. After association, each model is updated for the matched detection and the model probabilities are recomputed from the measurement likelihoods. The pairing is complementary by construction: the constant-velocity model estimates linear motion cheaply and well, and the constant-turn model covers the rotational case it cannot.

Life-cycle management handles the gap-versus-noise tension with two guardrails. An unmatched trajectory keeps predicting for a bounded number of frames before it is terminated, because an unmatched track usually means occlusion rather than disappearance, and restarting the trajectory later would fragment it. An unmatched detection has to persist for several consecutive frames before a track is born, which filters out occasional false positives. The two rules guard opposite failure modes.

The priors are where we departed from convention. Rather than setting the switching matrix and the covariances from experience, we treat them as an optimization problem and fit them with particle swarm optimization. The name invites a misreading worth clearing up: particle swarm optimization is a derivative-free search over parameters, it has nothing to do with particle filtering, and no particle filter appears anywhere in this work.

Two caveats on that fitting, and we would rather state them than have them discovered. The optimization runs offline and per dataset, so it is not online adaptation; the parameters are fitted to the data they were fitted on. And a symmetric switching matrix does not by itself tell you that vehicle motion in a dataset favors the constant-velocity model, even though it is tempting to read it that way.

View full-size image ↗

Bridging a missed detection is often cheaper than detecting it.

What we take from this

Latency budgets are a design input, not a measurement you take afterward. Once the real-time threshold is crossed, the remaining budget belongs to accuracy, and the architecture should be organized around that ordering.

A local receptive field is a cheap substitute for perfect calibration. When two sensors are aligned imperfectly, learning the correspondence in a neighbourhood beats assuming a pixel.

Reduce the dimensionality of the problem before optimizing the network. Pillarization is the enabling trick, not an afterthought.

Hybrid representations outperform pure ones here: learned appearance features, an explicit geometric projection, and an explicit state estimator, each doing what it is good at. The modularity also helps when something goes wrong, because each component has a distinct role and a distinct failure signature.

Complementarity beats tuning. Two models that fail in different regimes are better than one model that is mediocre in both.

And when the priors genuinely cannot be guessed, search for them. The admission that empirical IMM values may not transfer is the most useful methodological statement in the paper.

What it does not do

The honest summary is that the system is competitive in accuracy rather than leading. It improves substantially on detection speed and remains behind the best LiDAR-only methods on accuracy, which is the trade we chose.

The IMM-UKF has trouble with objects whose motion direction changes little. A filter that switches between hypotheses needs observable maneuver to be worth switching for.

Detection range is bounded by the camera field of view, which is inherent to this fusion design. Sparse distant objects fail, as the three-to-five-point example shows. The tracker is a separate stage from the detector rather than jointly optimized with it, which is the direction we point to for future work, along with improving accuracy without giving up the frame rate.

Closing

What we built integrates 3D detection and 3D multi-object tracking into a single pipeline measured against the real-time requirements of highway driving: a detector that produces oriented boxes efficiently, and a tracker that follows objects through uncertain motion using association, an interacting multiple model filter, and tuned parameters. The claim is not that it is the most accurate system available. It is that the accuracy is real at a frequency the fusion alternatives cannot reach, and that on a highway, being two metres behind the world is its own kind of inaccuracy.

Multimodal Learning | Chang Nie

MID: A Self-Supervised Multimodal Iterative Denoising Framework

The clean version often does not exist

Why the existing answers are conditional

What makes it hard

Restore by adding noise

Two networks, two different questions

Denoising is a way of framing a problem

How it is built

What we take from this

Where it stops

What kind of model this is

3D Object Detection and Tracking Based on Lidar-Camera Fusion and IMM-UKF Algorithm Towards Highway Driving

Perception that is blind for two metres at a time

What the fusion families cost

What makes it hard

Attach a neighbourhood, not a pixel

Collapsing the height axis to get the frame rate

Two motion models, and a search for the priors

What we take from this

What it does not do

Closing