Controlling the robot arm with C#...

This video is rather busy so I'll explain what's going on.

The C# application (called 'Project AM.E') is the main application that will contain all the functionality to start the webcam, analyse the picture for colour bricks, AI to sort the bricks and control the robot arm.  The Microsoft LifeCam window is only there so you can see the robot arm moving.  At the moment the application is just displaying dummy data and moving the arm as a first test...

• As the application loads it switches on the webcam (so the robot can see!)
• I press the 'start button' which currently just simulates analysing the picture
• As the robot arm starts to move I bring the Microsoft LifeCam window back into view so you can see it moving
• Notice the log at the bottom of the application screen which is showing what's going on throughout

First test controlling the arm

First draft of the GUI ...

The basic elements are there, the webcam view, the interpreted brick colours and on the right will be the sorted bricks.

I've added a listbox to display the debug information which looks rather good while it is working.

Next is to start controlling the robot arm and coding all of the functionality together.

First draft of the Graphical User Interface v0.1

No disassemble Number 5!

Anyone remember Short Circuit?

Yes, well that shows your age....anyway, one tip when building the robot arm, empty each packet for screws one-by-one, count them (which is the only way to recognise the subtle differences between them), then label them with their P number.

 Page 5 of 28

I think my cunning plan might have a hole it in, I'm not sure the pincers will open wide enough to pick up the webcam - so instead I might have to mount it on the yellow plastic shroud at the top....

And... finally page 28 of 28!

Number 5 - http://www.imdb.com/title/tt0091949/

The Robot Arm's here ...

I best get some Super Glue and a large hammer!

ROBOTIC ARM USB!

Errr, oh oh, look at all these tiny bits

Ikea has nothing on this - more beer needed!

Creating 'flight paths' for the robot arm...

To keep the logic simple when controlling the robot arm I've decided to create the concept of 'flight paths' as in predefined routes the arm will take rather than having a different route for each sort action.

So to move the unsorted green brick in position 0 into its sorted position 3 I would need to call the following predefined 'flight paths':

A - to move the arm to the default unsorted position
B.n - where n is the unsorted bricks position, 0 to 8
C.n - where n is the unsorted bricks return to default unsorted position
D - to move the arm to the default sorted position
E.n - where n is the sorted bricks position, 0 to 8
F.n - where n is the sorted bricks return to default sorted position
G - to move the arm to the default unsorted position
Either sort another brick or H - to move back to the arm start position

I could optimise the last brick sort step and move the arm directly from default sorted position to the arm start position, i.e., missing out G and H for the last sort and call it step I.

I'll also need a set of flight paths to pick up the web camera, take the picture and return the web camera before stating off sorting the blocks.  More code!!!

Robot arm 'flight paths'

Getting the webcam working ...

This was a lot more complicated than I first thought, there is example code on the Internet to capture images from webcams but not all of it works on Windows 8.1 64 bit with Microsoft Visual Studio Express 2013 Desktop, C#.

To cut a long story short I found that you need to use Windows Image Acquisition (WIA), which uses the Windows Driver Model (WDM) architecture.

Again this is just another piece of test code which will need stitching into a main application but what is does do in only a few lines of code is:

1. Switches on the webcam so you can see the live video feed from the webcam
2. Enables you to take a picture and automatically save it as a JPEG on your hard drive
3. Switches off the webcam 

Webcam video and image capture

Using the example code that Samuel dos Anjos has written (link below) I simply added a new button 'Take Picture' to the Form and added a takePic function to the userControl class.

Calling the new Save Image functionality

The video feed is displayed via a PictureBox (ImgWebCam) control in Samuel's example so all I needed to do is use that control to save its current image as a JPEG.

Saving the current image as a JPEG

The earlier code can then pick up that JPEG and start analysing it.  The fact that you can see the live webcam video feed as the Robot arm is moving the webcam to take the picture will I'm sure look pretty cool!

Some useful example code from Samuel dos Anjos:

http://code.msdn.microsoft.com/windowsdesktop/Webcam-Capture-50295c7f#content

Colour detection code...

Fantastic!  The original code was able to find 'blue' within the picture, so the next step is to search a specific region (2) only.  I think my detection will essentially check for red, green, grey, yellow and brown and depending on which colour it finds it will report that colour.

Search for the colour blue within the entire picture

This is the code which currently scans through the entire picture, so in theory changing the for loop to region 2's coordinates should do the job.

Original code

It seems to work, I tested it with another image that does not have blue at those coordinates and it didn't detect blue.

Modified code to test

The next and more time consuming activity is to write code to scan each region and identify the colour within it.

Testcard and region masking ...

The colour detection looks like it will be quite tricky so lets tackle it first.

I've created a 'testcard' photo using the actual webcam suspended over the Lego bricks, I'll get the Robot arm to pick it up and hold it when the picture is taken to again try and get a fixed point of reference.  The natural / artificial light is going to be a problem however I think there is a light on the Robots arm so I'll use that to keep the light consistent.

Photo of the Lego bricks from the webcam 1280 x 960

Hmm even at maximum resolution with lots of light the picture quality is not great.

Next I had a go creating a 'region mask' over the top of the picture so that I can start to look at specific pixel areas where each Lego brick is, from 0 in the top left to 8 in the bottom right.

Masking out the regions on the picture where the colour detection needs to analyse

Using Google again I've found some example code that analyses a bitmap picture and detects if a selected colour is present.  What I'll do is try and modify it so that rather than searching the entire picture it searches specific regions i.e., 0 through to 8 to see what colour brick is present.

Calculating region 2's coordinates using Microsoft Paint

I'm going to use some example C# code by Hemant Srivastava which detects a colour within an entire image, it managed to find the blue colour so lets have a go detecting the colour in region number 2 by modifying the code to only scan within region 2's coordinates.

See this link for Hemant's example code: 

http://hemant-srivastava.blogspot.co.uk/2012/11/image-color-detector-in-c.html

Google and the Internet are your best friends...

While I'm waiting for the Robot Arm to arrive in the post I've been looking around for inspiration for the C# code I'll need to control its movement and I've come across a really great piece of ingenuity from Matthew Dally who has captured the USB signals sent to the Robot Arm from the supplied Windows application. He's taken the codes and written a handy wrapper in C#.

Take a look...

http://matthew-dally.blogspot.co.uk/2012/08/c-5-axis-robot-arm-code.html

 (C) Matthew Dally - C# 5-axis Robot Arm Code

Defining some constants...

OK so, this project could get complicated very quickly if I try to handle every eventuality, e.g., bricks in different places, unlimited colours, different shapes etc.

So the golden rule of this story is KEEP IT SIMPLE

Fixed points of reference.....

This feels like the best place to start- on the top left of the picture are the 'slots' where the Lego bricks will be placed.  To the right are the locations where the sorted bricks will be placed.  I think I'll go with a maximum of three colour piles as I don't want to break my new golden rule only 30 seconds after I created it!

The ARM position is where the robot will sit.

Creating some fixed points of reference - to help me and my robot out

The start state with the Lego bricks in their positions ready to be sorted

And now the sorted bricks - just the middle step to go then!

Getting started with the idea...

My intention is to:

1. Use the webcam to look at a 3x3 grid of randomly coloured Lego bricks, the position of the grid will always be the same to help keep the robot arm movement simple.

2. Identify the colour of each Lego brick using the webcam; I'm thinking about using the webcam to take a bitmap picture and look at each bricks location in tern to determine its colour.

3. Determine which colour pile that brick should be put into using C#

4. Use the robot arm to pick up each block and put it into the correct coloured pile

The component parts...

USB Robotic Arm Kit from Maplin 

Microsoft LifeCam VX-3000

Microsoft Visual Studio Express 2013

Getting started

My Robot Project!

Hello!  My goal is to have fun creating a project to sort Lego bricks into piles of different colours using a robot arm, a webcam and C#.  I haven't written any code in the best part of 10 years and have never written anything that controls peripherals so it might be quite challenging.  That's the fun bit!

I've decided to keep a blog which will be a useful reference for me looking back over the project to see where improvements could be made and also if anyone else wants to have a go.