Creative thinking

One thing  that I’ve been paying special attention to is creative thinking.

I’ve long had this theory that it is merely logical thinking, after taking a step back from the problem at hand, and looking at it at a space on level above, or a meta space, or perhaps adding an extra dimension into consideration.

I’ve been paying more attention to it in the interviews I conduct at my company, and in my problem solving – to turn hypothesis into firm belief.

For dance, taking a step back would mean instead of executing the sequence of moves, looking at what rules those moves are made out of, and then altering those rules. You’d get drastically different moves! And that tends to be interpreted as being quite creative.

For music, there are some straightforward illustrations as well. A piece of music can start to feel quite flat and lifeless if you’re just playing interesting patterns in a scale. Perhaps you find one interesting pattern; then another interesting pattern. And then eventually you exhaust that variety of new note patterns. You’re stuck playing by the rules of note patterns. If you take a step back and consider what the sound is made out of – the properties of each note, or the notes as a system – you can start to vary the music at that level. Instantly, a dramatic change in the quality of music: staccato, legato, attack, volume, string bending – the music communicates in many dimensions other than tonality.

For problem solving, taking a step back from the known rules, and reflecting on the nature of the rules – tends to produce sufficient variety to solve the problem. Within the rules of the problem, you are playing in a formal system. But the rules themselves are described in a formal way, so they are also a formal system. So while varying them dramatically changes the universe of someone stuck playing by those rules, it’s done by the same means as exploring those rules – and so what feels like a creative leap to someone stuck inside the box, is merely logical thinking for someone who is able to think at the meta level of the problem.

One example immediately available to me is a system I designed at work.

We have academic papers submitted to our website as pdfs, and some of them are spam. After one is submitted, we eventually run a classifier to figure out whether it’s spam or not – in a background job on a dedicated job processing server. The classifier takes into account many signals, including whether the user is spam or not. So every now and then we re-run the classifier, to catch tricky spammy works littering our site.

Additionally, we have a smart algorithm that discovers academic papers related to a particular paper.

We wanted to notify users when a paper is uploaded to the site, if it ends up being related to a paper they recently downloaded. But only if it’s not spam.

There were two obvious designs, which I think immediately stem from the description of the problem, and the stages presented:

  1. After the paper has been classified as spam/not spam – run the related papers algorithm, but only if the paper is freshly uploaded. We don’t store this sort of data, so we’d need to find a place in our database just to store this “freshly uploaded” flag, just for this purpose.
  2. After the paper has been uploaded, start a job that will wake up every now and then, and check if the paper has been classified as spam/not spam yet, and run the related papers algorithm if it has indeed been classified.

Both approaches have undesirable qualities. Storing a “freshly uploaded” flag is fickle and ugly. Spinning wheels actively asking the system whether it’s done classifying a paper is fragile and feels like unnecessary work.

So I proposed a third solution:

  • Have a system that accept events about papers. It can have many different kinds of events – paper uploaded, classified, deleted, commented on, recommended, bookmarked, downloaded – and can react in various ways to those events. Each event is stored in a database, and we have a log of everything happening to every work in the system. Reacting to an event can involve looking through this log.

This solves both of the previous problems. The “freshly uploaded” flag is automatically stored in the system in a nice, non-ad-hoc way – as the recent “paper uploaded” event that the system has stored. If that event is stored, and we determine that it hasn’t yet met its “paper classified” event, and a “paper classified” event comes in, we run the related works algorithm.

It’s not clear where this last solution comes from. It may feel like a creative leap. The first 2 solutions only used terms from the description, but this solution includes brand new words and concepts.

The way I derive it, however, can be explained with my framework of creative thinking as logical thinking, after taking a step back into the meta-space of the problem.

Here is the sequence of thought:

  1. In one solution, the data I need to make my decision is not in one palce
  2. In the other solution, I’m not reacting to changes in the system at the right time
  3. There are 2 places that exist that make sense to introduce the solution into, and they’re both bad
  4. Maybe there needs to be a new place?

This changes the question from “Where do I put my logic?” to “What do I build?”. That’s the stepping back into the meta space part. The rest is straightforward logic:

  1. I want the data to be in one place.
  2. The data is “paper uploaded” and “paper classified”
  3. What do those two bits of data have in common? They’re both “(noun) (verb past tense)”, which could be called an “event” in the abstract
  4. So I can make a place where events go, and direct my events there

And beyond that, it’s a straightforward matter of figure out how that data should look, where I am again playing within the rules, and don’t need a creative leap.






Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s