Reaction test: Windows 10 Insider Preview vs Mono vs NetMf vs Arduino

Archive
1

Hi!

i wanted to know how fast the Raspberry Pi 2 with Windows 10 and an Universal App will react on a signal on an input port. I built a test scenario where an arduino is the stopwatch and a device has to react as fast as possible to a falling edge from an input port (generated by a manual push button) with a low signal to an output port. This falling edge will receive the arduino stopwatch and it will stop the measurement.

I measured some other configurations, too. Look at the image in the attachments. All values are in micro seconds.

The “Ausreißer” column means that there were rarely some measurements that were very high. Maybe because the operating system has done another task at the same time.

In the “Schleife” columns there are the results from an endless loop code whereas in the “Interrupt” columns the results from interrupt or event based code are.

Of course, the Arduino was the fastest - with the least powerfull processor. :slight_smile:
I think it’s because there is no operation system and no other things that will slow done the execution.

It is good to know how long the reaction time is with the different technologies. Even the slowest reaction time of 0,026 Seconds may be fast enough for most applications - but not for everything.

Keep in mind that i tested the Insider Preview of Windows 10. The results will not show the performance of the final Windows 10 version!

I made a more detailed posting on my page:

Hope that helps someone.

Bye,
Daniel

Update 2015-08-15: Today I measured the Pi with the final Version of Windows 10. It is eight times faster then the CTP-Version if you poll the ports directly. :dance: I updated the image, too.

11 Likes
2

How fast was the arduino running managed code?

3

@ Daniel7 - this is an unfair comparison. This is like saying let’s see how a space rocket can do 0 to 60 to compare to my car…oh my car is faster…Maybe a bad example :slight_smile:

1 Like
4

Well, it’s not necessarily unfair, if what you’re looking for is near-instant reaction to input.

It’s definitely apples-to-oranges, though, comparing the strengths of one platform to the weaknesses of another.

Let’s see it implemented in RLP :wink:

1 Like
5

@ Daniel7 - For your Cerbuino interrupt time were you measuring system time when the event was raised or were you reading the interrupt time argument from the event? It looks to me like you’re probably reading system time since it is divisible by a 20ms time slice. Granted, depending on your requirements this could be a valid measure but it may not tell the whole story.

1 Like
6

Hi!

of course you all are right. :slight_smile:
It’s not a fair comparison and i think Gus’s example from the rocket and his car is very valid. :slight_smile:

My question was not which device / technology is the fastest and gets the golden pineapple trophy. It’s more an answer to my question if the RPi / Cerbuino would be fast enough for getting a reaction to an input signal for using it for a specific application. And the answer for my application is: yes, it is. slower than the others, but it is fast enough.

Gus, i think if the challenge is to travel to the moon, you should choose the rocket even if the car would be faster in 0 to 60. :think:
But maybe you should take both because you have to travel to the rocket airport first.

RLP would be a nice idea. I should test this, too. But i think NetMF devices are more associated with C#.

ianlee74: the Cerbuino does not measure anything. I attached a screenshot of the cerbuino source code.

Daniel

7

The first question from almost every hardware guy I talked about NETMF was “how fast can it toggle an output pin?”. I think it is some sort of universal intergalactic metric, suitable to measure pretty much anything. So this test is actually quite valid and accurate. Because, you see, hardware guys do not care about building firmwares quickly and painlessly. Wasted time is not important. Wasted cycles are…

2 Likes
8

I’d say this purly depends on the application.
I have for example an data processing application, runing on a G120.
It does not matter if the time in which I react on the data Input, process it and send it out again is constand or incredibly fast.
All I Need to make sure is that the time stays below the cycle time.
The reason why I choose an NETMF Controller and not an bare mettal one here is that the application is quite complex. There is alot of code which handels a lot of data (uses about 3MByte of RAM for data).
It was way easier to implement it in C#, even that I know that I waste tons of CPU cycles and Memory allocations, compared to an bare mettal CPU.
Time to market was the main driving reason to choose a NETMF device.

1 Like
9

@ Reinhard Ostermeier - Well, obviously you are not a hardware guy! :slight_smile:

10

@ Simon from Vilnius - Originally I’ve been, but not electronics. electric only.
Later I got an SW guy.
But I also have NETMF applications, where Timing is more important.
Recently I moved repetitive SPI readings from an ADC to RLP, so I could increase the sample rate from around 2kHz to 40kHz on a G120, which does Networking, CAN, UART and other stuff in parallel.
But of course, for the Hardware design I have my Colleges.

11

[quote=“Daniel7”]
ianlee74: the Cerbuino does not measure anything. I attached a screenshot of the cerbuino source code.[/quote]

Well…actually, it does but you are not taking advantage of that. As I suspected, your measurements are based on the time of the event and not the time of the interrupt. Take note of the “data” argument being passed to InputPort_OnInterrupt() which tells the precise time the interrupt was received. If your objective is to measure how fast an event is raised and you can respond to it then you’ve done the test correctly. If your objective is to measure how accurately you can measure interrupt times then it is flawed.

1 Like
12

Hi ianlee74,

thank you for the clarification. I used the data parameter in an other project. It is very useful. Maybe the Cerbuino would be a better stopwatch than the arduino. :slight_smile:

For this project it was only relevant to respond as fast as possible.

Daniel

13

@ Daniel7 - If you want to measure how fast the board is reacting, the data Parameter timestamp is the correct one.
If you want to measure how Long it takes before you can actually do anything, the current System time in the Event handler is more accurat.

1 Like
14

Thanks for the data. It is helpful to know the absolute best response time for an interrupt in a given application. But the whole chain of events for the interrupt needs to be understood.
Regarding NETMF Interrupts.
NETMF will buffer interrupts, hence the need for the “data” with the time stamp. The problem is if the interrupts continue faster than can be processed you can run out of memory pretty quickly on the Cerbuino. It takes longer on the G120 and G400 but it will eventually happen. A while back I run some test with a frequency generator connected to an interrupt pin and periodically checked the memory usage. (see code below) I continued to raise the frequency until the memory usage started growing. Using that I was able to find a rough interrupts/sec for each module. Unfortunately I can’t find my results.

long AvailMemory;
 
Debug.EnableGCMessages(false);  //Turns off Garbage collection debug messages
 
AvailMemory =  Debug.GC(False):  //Returns the available memory.  AvailMemory decreases as the queued interrupts accumulate.

I don’t know how Windows 10/Linux/Android handles the interrupts when the occur faster than the OS can process.

As with any discussion about deterministic response or “real-time” there has to be some definitions given and time limits need to be defined. And of course those are all application specific. It does help to know the basics before attempting to create a product. Just looking at the data given in the first post I probably wouldn’t consider Windows 10 on the Pi2 for the flight controls on a quad copter. But it would do just fine if I was doing a home automation project.

2 Likes
15

That’s cool that you took the time to put this together.

Please keep in mind that Windows 10 on the Pi2 is far from complete. Any timings you have (or post) should be qualified in big red letters with the fact that the software is preview/pre-release and not final. :slight_smile:

I’m not asking that this be taken down (nor can I give permission to leave it up, should you ask), but the license you agreed to when you downloaded Windows 10 for the Pi2 does technically prohibit disclosing benchmarks on pre-release Windows 10:

[quote]Are there things I’m not allowed to do with the software? Yes. Because the software is licensed, not sold, Microsoft reserves all rights (such as rights under intellectual property laws) not expressly granted in this agreement. In particular, this license does not give you any right to, and you may not:
i. use the software in a live operating environment;
ii. disclose the results of any benchmark tests of the software to any third party without Microsoft’s prior written approval;
…[/quote]

If up to me, I personally would leave it up, but with the big/bold caveats about being pre-release software.

Pin toggling and interrupt handling are important tasks for sure. However, what people often miss when looking at benchmarks like this, is the higher level stuff you give up. For example, the Arduino is spectacular as a low-level microcontroller. But if you want threading or other higher-level concepts, or hardware abstraction, you can argue that NETMF does that better, even though it may be slower at toggling a pin.

Although the price might make some think otherwise, the Pi is also more of a Big Brain in a Big Brain/Little Brain pairing. If you want to do something like fast PWM, Analog IO, or fast flipping, adding a HAT with a little brain on it is often the better choice. This is exactly the type of thing GHI is working on.

A hardware abstraction layer and set of drivers are between you and the pins on the Pi with Windows. It’s for very similar reasons that every PC you own includes dedicated microcontrollers on the motherboard for handling other functions. (Tablets include touch processors, for example).

Through RLP in NETMF, GHI also has ways to make pin toggling/handling faster if that is a key need for your solution, so you get the best of both.

With garbage collection and the buffered nature of NETMF, getting an absolute performance benchmark will be difficult. Because of the way we take care of other things, and offer high-level constructs not normally seen on microcontrollers, you are a bit further removed from the hardware than on an Arduino.

Finally, also keep in mind that we’re working on NETMF performance as well. Our roadmap for NETMF includes a number of things that will enable it to run on lower power processors, as well as just perform better. Maximum performance was never the main focus of NETMF, but rather it was about the productivity of .NET on a microcontroller, with the ability to port to many different underlying architectures – something arguable more relevant then than now.

Pete

5 Likes