What is Imaginary Programming?

imaginary-dev, or: How I Learned to Stop Worrying and Love GPT

As programmers, one of the first things we are taught is that while computers don’t always do what we want them to do, they generally do what we tell them to do. We soon discover that most of the pain in programming comes from aligning what we told a computer to do, and what it actually does. Don’t get me wrong—I enjoy being very precise r and giving computers specific instructions to generate an output. But sometimes, I wish the computer would just figure out the messy details for me. The former is quite easy in current computer programming languages; the latter is quite difficult.

This is where Imaginary Programming comes in.

Imaginary Programming? What?

Imaginary programming is the idea that we can harness powerful new artificial intelligence models to implement human cognition tasks inside “normal”, deterministic code.

Let’s use an example. Let’s say that you work as a full-stack developer at a company where users pay you to create new database instances and populate them with data. One day, your designer comes to you, and the following conversation occurs:

Designer: “We want to add a feature that, whenever a user creates and names a new database table, we suggest five column names that might be useful for that new table.”
You: “Great. What would you like the column names to be?”
Designer: “It would be ideal if the column names could be appropriate to the table name, just like… the five most likely columns that the table would have. So, for hotel rooms, it would have room ID, obviously, but also maybe things like how many beds the room has.”
You: (starting to worry about the sprint schedule and your basic ability to complete this task): “Uh… um… ok.”

There are, of course, some ways we could go about solving this with traditional programming. It would probably involve going through your existing customer data and doing some analysis of the most common column names or hard-coding in some common column name patterns like <table name>_id. This would sort of, kind of, maybe work, but it certainly wouldn’t give your user the magical experience your designer envisions where every possible table name gets a bespoke set of column name suggestions.

If you turn to Imaginary Programming, you might be able to solve the problem in the way your designer imagines. To start an Imaginary Programming task, you write out a function prototype of what you want to do, along with a descriptive comment explaining what the task at hand is. For this example, it might look like this:

/**
 * This function takes in a name of a new database table and
 * suggests 6 names for columns that the user may want to add to
 * the table.
 *
 * @param databaseTable - a name of a new database table that
 * a user is creating
 *
 * @returns an array of good column names that the user may wish
 * to add to the new database table
 *
 * @imaginary
 */
declare function getSuggestedColumnsForDatabaseTable(
  databaseTable: string
): Promise<string[]>

Notice a few things about this code that are important to understand:

This is well-typed, valid TypeScript that defines a function.
The TsDoc comment is fairly descriptive of what it does.
There is an @imaginary tag in the TsDoc comment.
There is no implementation.

If you’ve already set up your project to use Imaginary Programming with imaginary-dev, there is no next step. You can now call the method you defined and test it out to see how well it works. It really will return values as is. Imagine that this is your user input handler for your user interface when the user makes a new database table:

async function onCreateNewDatabaseTable(newTableName: string) {
  // call imaginary function to get good suggested column names.
  const suggestedColumnNames = await getSuggestedColumnsForDatabaseTable(
    newTableName
  )
  populateSuggestedColumnNameDropdown(suggestedColumnNames)
}

Note that you are calling a function, getSuggestedColumnsForDatabaseTable, a function you never implemented and never intend to implement, and it just works. You’ve imagined it into existence.

After playing with this for a while, suppose you decide that you want to suggest not just the column names to the user, but also potentially their SQL types. All you need to do is alter the imaginary function like this:

/**
 * This function takes in a name of a new database table and
 * suggests 10 names and SQL types for columns that the user
 * may want to add to the table.
 *
 * @param databaseTable - a name of a new database table that
 * a user is creating
 *
 * @returns an array of good column names and standard SQL
 * types that the user may wish to add to the new database table
 *
 * @imaginary
 */
declare function getSuggestedColumnsForDatabaseTable(
  databaseTable: string
): Promise<{ columnName: string; sqlType: string }[]>

All you have done here is change the function comment to note that both column name and sql type are coming back from the function now, and you’ve also changed the TypeScript return type to an array of structured objects with column names and sql types rather than an array of strings.

Once again, you haven’t done anything to the implementation of the function, because the implementation does not exist. Despite that, it works. Your designer loves it, and you are a hero.

A look behind the curtain

Ok, we need to pause this story because, frankly, what on earth is going on here? How are you able to call a function that, for all intents and purposes, does not exist? As you might expect, some really fun things are going on underneath the covers here.

First, imaginary-dev provides a plugin to the TypeScript compiler that scans your codebase for functions marked with @imaginary and replaces those functions with a call to imaginary-dev’s runtime engine. The imaginary-dev runtime engine calls into OpenAI’s GPT, a large language model (LLM) that we use as the implementation engine for your imaginary functions. In essence, imaginary-dev starts a mini-dialog with GPT at runtime that looks something like this:

imaginary-dev: “Hey GPT, if there were a function called getSuggestedColumnsForDatabaseTable that gave 5 suggested column names and types for a new database table name, and if that function was called with the string ‘hotel_rooms’, what would that function return?”
GPT: “Probably something like hotel_room_id (bigint), num_beds (int), floor_id (bigint), hotel_id (bigint), and num_baths(int)”.

GPT is surprisingly good at coming up with answers to questions like this. Large language models have been trained on enormous corpuses of text, and they are quite good at all sorts of text-based tasks. The responses that GPT gives are sometimes a bit conversational and difficult to parse, though, so imaginary-dev does a fair amount of tedious work to make sure that GPT’s answer can be correctly massaged into the TypeScript data structure that the function in question is supposed to return. As a user, you don’t have to worry about it, though: just specify the TypeScript return type you’re looking for, and imaginary-dev will make sure that GPT returns it.

Think of TypeScript as the right tool to deal with the deterministic, structured, known world of data and algorithms, GPT as the right tool to deal with the messy, unstructured, unknown world of human language and thought, and imaginary-dev as the tool that bridges the gap between the two.

This is not Copilot

A lot of programmers, when first exposed to Imaginary Programming, think that it’s the same as Github’s Copilot, another excellent programming productivity tool based on GPT.

It’s important to understand that Imaginary Programming is not like Copilot.

The difference is that Copilot is all about generating deterministic code for you at coding time that will be inserted into your source files and run in a well-defined, deterministic way at runtime. Imaginary.js does not generate any code to implement your function; rather it calls into GPT at runtime to ask for what the answer would be with the specific arguments being passed in, if the function in question existed.

Copilot is very cool, but it addresses a different problem than Imaginary Programming. Copilot lets you code the things you could already build faster; imaginary-dev lets you code things that previously would have been impossible.

What is newly possible with Imaginary Programming?

Frankly, we haven’t yet figured out all the things that Imaginary Programming excels at, because there aren’t enough developers using these technologies in creative ways yet. We do, however, know a bunch of areas where it currently does well:

Suggest answers automatically for the user rather than asking them:
- “Give this Spotify playlist a good name based on its songs.”
- “Come up with a good emoji icon to use for the name the user typed in.”
Classification of text:
- “Tell me whether this comment is positive or negative about my product.”
- “On a scale of 1 to 10, tell me how angry each email in my customer inbox is so I can triage who I get back to first.”
Entity detection:
- “Find all the names, addresses, or dates in this email.”
- “Find the ingredients and their quantities in this recipe.”
Data and structure extraction:
- “Given this essay about Winston Churchill, tell me three interesting facts about him.”
- “Here’s a math word problem. Extract what the question is actually asking in a structured way.”
Natural language translation: “Translate this text to Spanish.”
Change the style or emotional content of text: “Make this SMS more supportive and encouraging.”
Summarization:
- “Summarize the chat I just had with the customer into easy bullet points that I can forward to the product team.”
- “Make this passage readable by a third grader.”
Reverse summarization: “Turn these three bullet points I want to convey to the customer into a friendly email.”
Conversation: “Come up with a good response in this chat, acting as a helpful agent.”

There’s undoubtedly much more to learn about Imaginary Programming’s power and capabilities as we get it into the hands of more developers and as the underlying AI models get better.

Everybody has their flaws

Imaginary programming is a truly magical experience when it works, but we need to be honest about its current capabilities. There are four primary drawbacks to keep in mind as you start integrating Imaginary Programming into your applications:

First and foremost, Imaginary Programming is not great at facts. Large language models like GPT know an enormous amount, but (at least for now) they can easily get confused on facts and deductive reasoning. Any task where you are asking an imaginary function for facts about the world or reasoning based on facts is probably going to be sub-optimal. Similarly, GPT knows nothing about mathematical reasoning, so it doesn’t make a lot of sense to ask it questions in that world.
Imaginary programming can get confused if you make the data structures very complicated. GPT is totally good with returning arrays of objects with well-defined required and optional properties. It also generally does fine with objects that have other nested objects as property values. However, if you ask it to parse a large document and extract information into a seven-layer nested data structure, you probably won’t see much success.
Imaginary programming cannot deal with large amounts of data. Large language models like GPT are very processor intensive, and Imaginary Programming only works when the combination of inputs and outputs is on the order of a few kilobytes. This will almost certainly change as the algorithms and GPUs running large language models improve, but for now, this is the limit.
Imaginary programming is susceptible to jailbreaks. Large language model prompts are susceptible to "jailbreaks". As a simple example, user-supplied arguments can succeed in telling the LLM to ignore its previous instructions and do something else, like reveal the contents of the other arguments to the imaginary function. We are currently looking at how to mitigate jailbreak attempts, but for now, be cautious if you are using user inputs that could be hostile. Think through what the possible repercussions are if the LLM answers incorrectly because a hostile user asked it to.
Imaginary programming adds some latency and runtime cost. Because every Imaginary Programming function call sends out a request to GPT, and since OpenAI is currently rapidly scaling up their capacity, solving problems with Imaginary Programming can add latency to your app. You will definitely need to throw a spinner up in the user interface if your user is waiting for an imaginary function’s results, but adding a half second of latency to do things that were previously impossible is a good tradeoff. Additionally, you need to pay OpenAI for the use of GPT on a per-call basis. Generally speaking, the rates are quite affordable (the most expensive GPT model is generally less than a penny per call), and there are ways to cut down costs if it’s becoming a real problem.

O brave new world, that has such things in it

For the last decade, artificial intelligence problems at web and app companies have been the domain of specialized machine learning teams full of people who know how to build and maintain specialized AI models. With the emergence of foundation models like GPT, everyday web and app developers are going to be increasingly able to incorporate AI into their projects without the help of machine learning or artificial intelligence experts.

This is the promise of Imaginary Programming and imaginary-dev: If you’re a frontend developer or a full-stack developer or a mobile app developer, you are now also an artificial intelligence developer. Without learning a complicated new discipline, your toolbelt is getting bigger, and you get to build cooler, more ambitious, more complicated projects–some of which you never would have been able to make before. I’m really excited to see what you build.