Behavior reconstruction: the next frontier of AI?

This post first appeared on AI Workplace. Subscribe to the newsletter to get articles and exclusive thoughts on the AI Workplace.

One exciting application of deep learning is image reconstruction: take an image with “holes”, and reconstruct the missing parts. The basic idea is that given enough complete images, a neural network learns the local structure of the images. When presented with an image with missing data, it fills in the missing areas based on the structure learned in past images. The results are impressive. Exciting in its own right, as this technology matures, it’s easy to imagine a future where we can recreate not just images but a person’s life. I call this behavior reconstruction, since we are attempting to reconstruct behaviors and experiences in a person’s life. 

The identities of a cormorant and its gull posse were protected, creating holes in an image.

This idea may seem radical, but we do it all the time. Whenever there is information asymmetry, we try to fill in the missing data. This happens whenever we make a decision: whether a defendant innocent or guilty; whether to rent an apartment to an applicant, whether to buy or sell assets, whether to lend money to a prospective borrower, whether to hire this candidate over that candidate.

Deep learning techniques can reconstruct an image with missing parts (simulation).

In the world of finance, prime brokers play a role similar to an Internet platform provider. Prime brokers act as a custodian and provide lending services to hedge funds and other investment firms. Due to the nature of the role, they tend to have a rich view of a client’s positions. When I worked in investment management (both buy and sell side), one of the interesting questions was the information asymmetry between the hedge fund and the prime broker. Hedge funds are known for secrecy and are like individuals that want to protect their privacy. On the other side, prime brokers want to know all the positions that the hedge fund has. Without this knowledge, it is difficult to assess the risk exposure the fund has and consequently how much risk the bank is exposed to when lending to them. But hedge funds don’t want to leak this information and thus have multiple prime brokers (creating irony in the process). Prime brokers know that hedge funds don’t use them exclusively, so the problem is how to construct a comprehensive, yet accurate view of the hedge fund with incomplete information. Hence, this problem mirrors that of image reconstruction.

Resumes, social media, and interviews are all collages of an individual’s experiences and behavior.

Now consider hiring. A hiring manager is presented with a resume and a cover letter (possibly). Throw in an interview, some social media stalking, and possibly a reference, and we’re supposed to choose “the best” one from a pool of candidates. Ultimately we are trying to predict the success of a candidate based on incomplete information, which is also biased. They are all patchwork collages of someone’s life: not a complete picture. Obviously a candidate represents more than just these artifacts. Behavior reconstruction would fill this gap to produce a complete picture of a candidate.

How believable could this reconstruction be? And even if it were believable, does that mean it’s a proxy for someone’s actual behavior? Would we trust a prediction based on the approximation? It begins to skirt Minority Report territory but is also common in statistics under the name of imputation. As the imputation gets more sophisticated, does it become more valuable or more dangerous? What do you think? Exciting or freaky, soon to be fact or forever fiction?