1

In this tutorial I am going to go over a brief history of artificial intelligence. Let's begin.

The Beautiful Equation
a0ac936120a8fa20eba9c2f4891766f5

OK I agree, it's a bit heavy to begin with but towards the end we will see why the above equation is, in my opinion, of great importance to the field of AI. But first let's go back in time to ancient Greece.

Ancient Greece
The Greeks dreamt about artificial intelligence, playing God if you will, they probably weren't the first. But the idea of mechanically creating an intelligence to rival our own has always been on the frontier of science. And it is no surprise why... the human brain is an enigma, an amazing feat of biological engineering, capable of making complicated deductions and understanding subtle nuances. Clearly, no hard coded computer program can rival such mastery.

So our immediate question is... how do we recreate this? Do we follow nature as an example?

Mankind masters flight
bd8194da70e35678f4d990fb5809bc9a Mankind has always taken their inspiration from nature. For a long time mankind was hellbent on being able to fly. In Greek mythology Icaraus forges wings out of wax to give himself the ability to fly. It is just myth of course, but man's idea to emulate nature seems a good starting point. Many of the first flying contraptions devised tried to copy the flight of a bird by copying the flapping of the wings to generate lift. But no such devices ever succeeded. Mankinds first powered flight came from aerodynamic wings that were FIXED but provide thrust to create lift.

So we could say flight was achieved my not completely copying nature but rather taking ideas from it and changing a few things along the way.

Should the quest for AI then try to COMPLETELY copy how the brain works completely? Only time can answer this question. But I believe it is a damn good place to start.

Next we are going to explore the brain.

The Brain
The brain consists of literally millions of neurons. Each neuron are inter connected with synapses. These pass electrical signals to the neurons. Now an individual neuron by itself is pretty useless. Learning occurs by the collective information of all the neurons in the brain.

Basically, when a neuron fires it needs to be excited and once it reaches a threshold it fires an electrical signal. Think of it like a reward system. If the dog behaves it gets a treat. So in the same way when a neuron is learning, if the desired action is considered good enough (i.e it goes beyond some threshold value) it fires.

d71f391fa54cfc48f49db1c11485f199

Now we can come back to our equation written at the beginning. All it does is describe the graph of a spiking neuron. The most important bit is that 'w' describes the 'weight' and in all neural networks learning occurs by continually adjusting the weights.

The downward slope of the graph is described by the exponentional function.

How do we replicate the spiking neuron graph?
So we have the graph of a neuron. Our next question is how might we replicate this? First we might think of this in software terms. You might write a program in say 'java' or 'C#' to replicate the graph. Now software is great but it is slow. What if we cloned the graph directly with hardware. Wouldn't that be faster?

Enter the FPGA
So what is an 'FPGA.' It is a 'Field programmable gate array.' Think of it like a bunch of logic gates that can be configured to how you like.

Why use an FPGA
Why? It's simple. FPGAs are fully Parallel. No shared resources mean no serial dependencies. Single Clock for cycle Numerical Iteration. One clock cycle equals one numerical iteration.

In short it is bloody fast and works much like a neuron. In parallel.

Problems with floating point calculation in hardware
Let's go back to our equation. Unfortunately, we've got exponential decay described in our graph. If you don't already know floating point calculations,or in our case differential/integral calculations perform horribly on hardware - (using logic gates).

So we employ a little trick which involves using the forward-euler method to approximate the exponential decay.

http://en.wikipedia.org/wiki/Euler_method

Let's not get too technical here, but essentially we use repeated subtraction to estimate exponential decay and remove the need for floating point calculations. Point to note, repeated subtraction can easily be constructed with logic gates.

So let's say we have now devised an FPGA for a neuron. The basic configuration to set up many in parallel will look as follows:-

5666ddc966609db3153f3420c51cf675

Conclusions
-Learning occurs by adjusting the weight matrix in a neuron
-A spiking neuron can be modelled with FPGAs which have the benefit of being massively parallel and having no serial depencies. This makes it much much faster than modelling in software.
-The forward-eular trick can be used to approxiate exponential decay thereby reducing the calculation computations.

Overall, this points us to the conclusion that modelling neural networks in hardware should offer more efficient and faster learning results.

Thanks for reading.

Further reading
If you are interested in getting setup, you can head over to http://www.xilinx.com/ and download their SDK. You can simulate your FPGA circuits with their software before buying the real thing (and flashing the hardware).

http://www.eetimes.com/document.asp?doc_id=1320753

Started window cleaning after uni. Got my first ladder, bucket and sponge and since then I haven't looked back.

2
Contributors
3
Replies
72
Views
3 Years
Discussion Span
Last Post by mike_2000_17
1

Learning occurs by adjusting the weight matrix in a neuron

Care to explain how? The logic required to update the weights is generally-speaking exponentially higher in computational cost than the actual neuron firing logic / dynamics, which is quite trivial to compute, even on an ordinary PC. One of the big problems with ANNs (Artificial Neural Networks, which is what you are describing, and btw, you should not use the general term AI as being synonymous with ANNs, because ANNs are a very tiny subset of AI) is the fact that algorithms to learn the weights of an ANNs are either (1) computationally intractable, (2) highly undercontrained, or (3) very crude and thus requiring a life-time of learning (which is the way animal brains learn).

The forward-eular trick can be used to approxiate exponential decay thereby reducing the calculation computations.

It seems to me like the whole thing could be done with an analog computer. This whole thing, in fact, looks like the dynamics of a fairly ordinary switched-capacitor circuit. I would imagine that with a few op-amps and transistors, this could be realized quite easily. If you need fast and instantaneous simulation of some dynamical system, analog computers are your friend, they've been used quite a bit in robotics to do some of the heavy calculations instantaneously. In fact, why not try this with a FPAA?

Also, if you are going to using forward-Euler, you might as well just stop thinking in terms of a continuous-time system, because it's not anymore. Discrete-time systems and discrete-time dynamics is, in my opinion, a much better way to analyse systems anyways, so you might as well go for that. I have found that systems can be very greatly simplified if you do the analysis entirely in discrete-time, instead of using something like Euler method to discretize it only at the end (which can also lead to some bad surprises).

In any case, from what I know, there is a pretty good concensus in the AI field that the actual mechanics of the neuron firing is not really relevant to the overall schemes (learning, performance, "intelligence", etc.). In fact, modern research in ANNs has departed quite a lot from anything that actually reproduces the exact dynamics of neurons, mostly because it's too complicated and it has been shown to provide no benefit at all, it's just one of these things where "nature" did it that way because that's just how it happened, but there are better and simpler ways.

Drawing from your analogy with Man's attempts to fly, reproducing the actual mechanics of neuron firings is the equivalent of flapping wings, i.e., it's trying to mimic nature while missing the principles that make it work. It's when people understood how lift was generated that they realized that the flapping wing designs were just overly complicated and that it did not make practical sense to reproduce that, so, instead they realized that they just needed wings with the proper shape to generate lift, and some means to propel the plane forward, and that's how the design was made.

the above equation is, in my opinion, of great importance to the field of AI.

I remain unconvinced. And, of course, if you ask what is the most important equation in the field of AI, the answer can only be Bayes Theorem, because any typical analysis of some AI method involves applying this equation about a dozen times over, it's the lens with which we see the world, and it's the single equation that defines intelligence. And the human/animal brains are just very efficient Bayesian inference approximators.

And given the vast number of basis functions (called "kernels") that are used in machine learning, whether it's for ANNs or support vector machines, you can hardly say that one is particularly more important than another. Typically, ANNs or SVMs are tested with several different kernels to find the best one to use, and the one you showed is not particularly great.

I'm curious as to why you selected a kernel that is exponentially decaying with time. Is there are reason why this time-component is important? We know that one of the limitations of neurons in an human/animal brain is the fact that they are very slow to respond (fire) due to there inherent electro-chemical dynamics, as evidenced by that graph that shows a 2.4 micro-seconds response-time, which is extremely slow from a computational point of view (which is in the order of nano-seconds). Do you think that this slow response has benefits? If so, which benefits? Why not just idealize the neurons into having an immediate response? I have some ideas why that could be important (e.g., Hebbian learning, Hopfield nets, etc.), but I'll let you develop on that.

http://www.eetimes.com/document.asp?doc_id=1320753

Why do you talk about modeling ANNs in FPGAs, and then link to an article about running some minimax routines on an FPGA? They are both methods under the very wide umbrella of "artificial intelligence", but they are very different and completely unrelated methods. They are not even in the same category of problems, as in, ANNs are traditionally for unsupervised learning / reinforcement learning, while minimax is in decision theory (finding optimal path through a decision tree).

Votes + Comments
nice
0

Thanks for the reply, let me just start of by saying I'm in no way qualified to answer some of your questions. It's way over my head.

I've just put together some thoughts/ideas when I looked over at modelling neurons in the brain.

So I'd like to pose some questions of my own.

In fact, why not try this with a FPAA?

Never heard of an FPAA can you explain. I just googled an article which cites

Field Programmable Analog suffers from serious noise and performance issues because the routing fabric contains a large number of transistors.

http://electronics.stackexchange.com/questions/25525/are-there-any-analog-fpgas

as being a major disadvantage.

In any case, from what I know, there is a pretty good concensus in the AI field that the actual mechanics of the neuron firing is not really relevant to the overall schemes (learning, performance, "intelligence", etc.).

Ah interesting take on things. Can we assume that modeling the neuron more closely and more accurately will make any difference to our understanding of intelligence? Or like you said is this unimportant or are we wasting our time. Until we cross that frontier I guess we don't know for sure.

Let's ask another question, our brain is considered having the capability of consciousness. Can an electrical system man made system ever be considered conscious? After all aren't the electrical signals that fire in the brain some sort of transistor.

What is the difference between biological transistors and electrical.

what are your thoughts on this?

Drawing from your analogy with Man's attempts to fly, reproducing the actual mechanics of neuron firings is the equivalent of flapping wings, i.e., it's trying to mimic nature while missing the principles that make it work. It's when people understood how lift was generated that they realized that the flapping wing designs were just overly complicated and that it did not make practical sense to reproduce that, so, instead they realized that they just needed wings with the proper shape to generate lift, and some means to propel the plane forward, and that's how the design was made.

Ah yes that is very true. But what happened when we tried to model the complex motion of flight by 'flapping the wings?'

http://www.youtube.com/watch?v=Fg_JcKSHUtQ

I'm sure you've seen this. Flight is possible without 'flapping' as in the airplane, but by our understanding that we can mechanically model lift by flapping of the wings unlocks another door of unexplorered possibilities.

Who is to say that the importance of modelling neurons as close to our brain as possible is largely unimportant?

What is your take on this?

Why do you talk about modeling ANNs in FPGAs, and then link to an article about running some minimax routines

Good point, I guess I was just trying to highlight the performance increases as opposed to the learning method.

Slightly off topic I also wanted to link to another really interesting article which I thought was a good read.
http://www.theatlantic.com/magazine/archive/2013/11/the-man-who-would-teach-machines-to-think/309529/

1

Never heard of an FPAA can you explain.

I had never heard of FPAA either, until I did a little looking up of some practical ways to implement analog computations. From what I gather, they can be used like FPGAs, but instead of programming digital logic gates, you program analog circuits. It seems like a nice way to make an array of analog computers in a small and efficient package. I can certainly imagine that there would be a lot of noise generated by all those transistors being packed on a single IC. But for an application like this (modeling neurons) the noise is probably inconsequential. If you were designing analog signal processing (which is generally aimed at removing noise or demodulating signals), FPAAs would not be suitable due to the noise level. But with neuron emulation, it is a kind of pseudo-digital system (due to the thresholded firings), meaning that the noise is not going to be a factor.

The key thing to remember with "the brain" is that it is a big approximation calculator. In other words, everything approximate, signals are fuzzy, and results are probabilistic. So, a digital logic system (like FPGAs or CPUs), whose main characteristic is the ability to make exact calculations, is an odd fit in the midst of it all.

Can we assume that modeling the neuron more closely and more accurately will make any difference to our understanding of intelligence?

Does your understanding of transistors help you understand how complex software is written? (rhetorical)

That's basically my take on this debate.

Until we cross that frontier I guess we don't know for sure.

As far as I know, this frontier has been crossed a while ago. I remember writing a neural network simulator when I was still in high-school, and at that, just reproducing work from decades before. There is a huge amount of work that has been done on many aspects of this, from closer to the neuron firing mechanism (like you described) to various network topologies and learning algorithms.

But in the realm of studying actual brains and how they work, one of the promising frontiers that is starting to be tackled is the ability to map out the entire set of connections in a brain and understand how the network as a whole functions. This is starting to become technically possible (map out neural connections). That is going to be VERY interesting.

But with the years of research on ANNs, things kind of went up and down a few times. At first, there was enthusiasm for reproducing the brain mechanisms with high fidelity, but it quickly became apparent that it was at best able to perform very trivial tasks in a terribly inefficient way, so it was almost abandonned as a result. Then, people started to apply them to more appropriate problems (sort of "fuzzy logic / control" problems or optimizations) and things picked up again, especially as people took out the high-fidelity neuron-firing mechanisms and replaced them by simpler and more efficient basis functions or kernels. But still, performance could not meet its competing approaches for these problems. And finally, the work in machine learning has come to be able to understand and explain what makes neural networks so powerful (in the brain), but still, ANNs are not the best expression of these mechanisms. Nevertheless, that understanding is starting to bare enough fruits in neural networks that people are starting to make serious attempts at making an "artificial brain".

But to me, one thing is clear, the actual firing mechanism is not particularly important. It's just the mechanism that happened to occur through natural evolution.

Can an electrical system man made system ever be considered conscious?

I see no reason why they would not be able to.

After all aren't the electrical signals that fire in the brain some sort of transistor.

Well, not that simple, but in essence, it's a simple electrical device, not unlike something you could make in a semi-conductor substrate. What is most challenging is the multiple input and multicast output nature of them, which is also a hint at where the actual complexity lies (as I already explained), that is, in the connectivity and complex interactions between them, not so much within them. As the IBM researchers clearly say, for example, is that the big challenge with creating a "brain chip" is routing all those signals (synapses) not making the artificial neurons.

Another interesting thing that makes a brain different from a hypothetical electronic system is the actual brain chemistry. We know that certain chemicals have effects on brain activity and it manifests itself in interesting ways like moods and concentration. I would imagine that an electronic brain would be much more in a constant "average mood", and I'm not sure it could have emotions, at least not quite in the same way as humans do. That will be an interesting to see.

But what happened when we tried to model the complex motion of flight by 'flapping the wings?'

It's a technical feat and a very nice machine, no doubt about that. But you still have a very inefficient system and it requires cutting edge materials to even make it possible. In other words, people's initial assessment was correct. The point is that a flapping wing design requires a lot of energy and can only carry as small weight. In that talk, they quoted 20-80W and a total weight of about 400g, which those not even match the efficiency of a quadcopter (which is one of the least efficient flying devices that people use). Birds in nature are mostly able to fly efficiently because they know when to soar, when to ride hot air pockets, and when to flap, and flapping is the last resort.

The thing to understand is that nature is a mixed bag. Evolution continuously optimizes things, and billions of years can work a lot of miracles. However, evolution is also limited in how much it can "leap" (in fact, it cannot make any leaps), and it is always bound to modify whatever it started with. So, when evolution has to turn a clumsy lizard-like creature, called a dinosaur, into a flying animal, called a bird, it managed producing something that works pretty well, but not particular remarkable by engineering measures. On the other hand, with all the wealth of life's diversity in water (where it originated in), it created organisms, called fish, that can move very efficiently in water.

but by our understanding that we can mechanically model lift by flapping of the wings unlocks another door of unexplorered possibilities.

Yes, of course, I'm all for exploring those avenues and it certainly makes for interesting research and lessons to be learned. But it's also important to be cautious about holding biological or natural things up on a pedestal as if they had some sort of cosmic "perfection" to them.

Who is to say that the importance of modelling neurons as close to our brain as possible is largely unimportant?

It depends on your objectives. If the objective is fundamental science, then pretty much anything is important as long as it's currently unknown or poorly understood (which, besides neuro-chemistry, I don't think is the case for neurons). If the objective is applied science or engineering, such as creating some artificial intelligence or brain, then what is important is what allows you to progress towards that goal, and in that vein, I don't think that further investigations of the details of neuron firing mechanisms are a particularly practical line of investigation, because we know that the real complexity lies elsewhere, in particular the network topology and learning mechanisms.

Slightly off topic I also wanted to link to another really interesting article which I thought was a good read.

I agree, it was an interesting read (albeit long and poorly written..). Cognitive science is definitely an important field that too often gets snobbed by the machine learning people. This is also another huge polarizing issue.

The machine learning folks would generally argue that once you get the learning mechanisms working, which are at the root of intelligence, the complex mechanisms of intelligence will arise. And they would often say that the cognitive science people are trying to fully understand how intelligence works in order to reproduce it, which they clearly see as a vain pursuit.

But then, the cognitive science people would say that they are trying to get at the root mechanisms of intelligence, while the machine learning are just coming up with method to optimize a poor and shallow imitation of intelligence (which is evidenced by Hofstadter's term "genuine artificial intelligence", which is merely a smear on anyone not doing cognitive science... yes, scientists can be petty too).

I do have to say that I find myself leaning towards the machine learning camp, but I don't denigrate the work done in cognitive science... but the lack of measurable progress in decades makes it a bit disappointing, to say the least. Cognitive science often feels more like a discourse in philosophy than a practical or applied science, as very well evidenced by that article you linked to.

Votes + Comments
Agree to change title to better describe content
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.