Gaëlle Dufour - Universal sorting robot

Universal Sorting Robot (2018)

👩‍💻 + TensorFlow + Embedded systems (Raspberry Pi & components) + 3D printing + Python

7 years ago, I was enthusiastic about the possibilities opened by the developments in AI, and notably image sorting algorithms such as the ones provided by TensorFlow. It is worth noting that at the time, I did not yet have access to LLMs and such to design the systems and write the code! So I decided to get hands-on and create a universal sorting machine!

A universal sorting machine controlled by a Raspberry Pi

The idea was simple:

objects would travel through a conveyor belt
a camera would take pictures or videos and classify the different objects on-the-fly using TensorFlow
a robotic arm would then grab the objects and sort them into bins

Of course, before this phase, there would need to be a pre-requisite step of training TensorFlow. This was what I actually started with, just to check whether or not the project was doable.

Use case: Sorting walnut pieces from shell pieces

This was an interesting use case because:

I actually had a real-life application as my family was making walnut oil at the time, and sorting walnuts and shells is extremely tedious and time-consuming
Industrial sorting machines were extremely expensive (hundreds of thousands), a Raspberry Pi controlled machine would literally cost a thousand times less
I wanted to test the capabilities of TensorFlow. I found this use case to be an interesting challenge for AI as the shells and pieces are almost the same color and their shapes are totally random. Up to that point, I had only seen machines that used AI to sort things based on their color or shape.

Step 1: Testing TensorFlow - a great surprise!

With a relatively small training process, TensorFlow was quickly able to sort things accurately. I was actually amazed! I had made 5 categories: coquilles int (external shell), coquilles ext (internal shell), brisures (walnut pieces), a decortiquer (needs more processing) and mauvaises (bad).

See below a few results from a new batch after my first training round:

internal shell with 99+% confidence ✅

external shell with 98+% confidence ✅

walnut piece with almost 100% confidence ✅

bad walnut piece with 99+% confidence ✅

I also found super interesting that the next "best guess" was pretty logical. For the shell, internal/external was the next best guess for external/internal. And for the good/bad pieces, the next best guess was bad/good.

Step 2: building the first prototypes, starting with cardboard

1 - Imagining the mechanism of the conveyor belt and the sorting bins

2 - Adding the camera chamber (right) and the Raspberry Pi connections

3 - Adding the first 2D printed pieces and the belt feeder mechanism (right)

Step 3: Local improvements

I eventually 3d printed the conveyor belt

Individual pieces were connected with a simple tiny rod

The matching cog was also 3d printed

This was the first version of the belt feeder that I came up with, it didn't work

And the final and functional version

The conveyor belt feeder goes to the very right of the system, elements can get out and onto the belt from both sides

The system in action with walnut shells

The process connected together and controlled by the Raspberry Pi: belt feeder, conveyor belt, pictures taking,...

Step 4: Soft robotics

I originally got the idea to push elements on either side of the conveyor belt... But the logic was flowed because the elements didn't arrive one by one, but in groups.

Before: the original idea was simple. It involved pushing elements to the side of the belt

... but it didn't work because the belt was too busy and I didn't have a way to make sure elements would come one by one

I realized that I needed a form of robotic arm. But my problem was that the shells and nuts pieces could really be of any shape, so it seemed to me too complicated to create a "hand-like" grabbing system that would involve a lot of computation. I finally stumbled upon this mechanism:

It is simply a ball filled with granular material that is pushed onto an object, then air is sucked out of the ball, and the object is tightly gripped (example)

I liked the simplicity of this solution, because then you only need to give it the position of the object to be grabbed. I unfortunately don't have the full mechanism in pictures and videos, but I did implement this with some success with a 12V vacuum pump, a medical air tube, and a balloon, pantyhose fabric and coffee grains!

The arm needs to move back and forth above the belt and above the bins to grab the objects from the belt and release them from the bin

The arm also needs to move up and down to grab objects

The prototype at the point where I left it

And... what happened to the robot?

Well, several things. First, my family stopped making walnut oil, so the application went out of the window. And I bought a 3D printer! And I just fell in love with printing and all my free time was from there on consumed by creating 3D printed objects.

3D printing became my obsession

That being said, it would be fun to pick up this project again. With the state of AI today, I imagine we would be able to make very very interesting things 🤓