Distributed Computing & Clustered Networking

Thanks to everyone that’s chimed in on my seemingly random questions over the past week I have the ability to create a dynamically clustered network of PCs and FEZ devices. :smiley:

Vault-Link not only networks the little buggers together, but with the addition of SBASIC 1.0 you can perform distributed computing!

I’m still running internal tests and adding more commands but I wanted to put out some feelers and see if anyone is willing to put together a coordinated effort to stress test the system by having as many FEZ devices as possible connect to my clustered network via the web at a predefined date & time.

Just let me know when & where…

I’m thinking early December but I want to see how many people I can get. With Hydra acting as the server I should be able to support over 100 client devices at once.

I can be in, depending on timezone challenges :slight_smile: I can certainly test out latency impacts ! :slight_smile: :slight_smile:

I can try and hook my spider up if that helps?

If I am around that time I will help you too.

@ Brett - Yeah it’s going to be tricky if we manage more people in the UK. Managing a 6hr time difference isn’t so bad, a 16hr difference can actually be easier, but managing both will be tricky. Hence wanting to work out the time in advance. :slight_smile: It’d probably be a Sunday morning or a very very late Saturday night, so if there’s any times that work for you let me know.

@ HughB - Every connection helps. I even have a windows app so if it comes to it everyone could double up connections (you can’t open multiple connections from the same device, the Vault-Link doesn’t allow it).

@ Architect - Awesome :slight_smile:

Is everyone in at 1 connected device each or do any of you have multiples you can connect? I can only get 2 on at once for now (1 ethernet port and 1 w/ WiFi), 3 if my Cobra II arrives in advance of the test date.

Is that only FEZ you’re interested in, or a Chipworkx + Mountaineer Ethernet would also be right ?

@ LouisCpro - Any and all NETMF devices are welcome and needed. :slight_smile: Hydra just happens to be the device I will have acting as server.

Well, let us know what it will deal with, and as Brett said, if the time zone is not a problem…

I can connect 4 devices.

@ LouisCpro - your timezone will be pretty easy to deal with as long a you don’t mind getting up early on a weekend :slight_smile:

@ Architect - Sweet!

The tests will be pretty simple. Load the app and run. Your device should automatically grab a seat on the network and be informed of every other device already there or that joins/leaves after. The server will ping the device at regular intervals. If I have time to add a GUI you’ll be able to select commands to send to any device on the network.

@ Skewworks - Cool. I have to make sure all my boards are up to date.

@ Skewworks - Or maybe it would be possible to launch the client before the server is up ?..If not, and regarding to the date, I will do the effort :slight_smile:

@ LouisCpro - there is a loop in the program to keep attempting connection…we could always just set yours to wait until a specific time before starting to try or give it a really long wait between attempts (like an hour) and you could just turn it on before bed and leave it.

@ Skewworks - OK

Count me in for at least one, possibly two, devices.

Excellent 11 - 12 devices so far. Let’s see if we can double that count before the 1st.

Maybe Gus should spin up 100 connections over at GHI. :wink:

What is Vault-Link?

Part of the PIP project Vault-Link allows devices (NETMF, PCs, Mac, Linux) to create their own clustered network to communicate with nodes. Each node is configurable and can optionally allow File IO, Hardware IO, SQLite, Distributed Computing (SBASIC, Skewworks Apps & EXEs on the PC side), Messaging, etc.

The network can be setup in multiple ways. The default method is UDP + TCP. When a device boots up in this mode it first checks UDP (for up to 5 seconds) to see if a server is broadcasting ident. If one is found the device will become a node if set to allow, otherwise it will drop off and report it couldn’t be a server. If no server is found, however, the device will assume the server role (again if allowed by config) and start broadcasting its ident.

As nodes join all other nodes are informed of the new addition and the joining node is given a list of available peers. The server regularly pings nodes for health checks and removes failed nodes (which is broadcast to all healthy nodes). Note health monitoring will NOT be present for this run since a failed connect hangs forever in the current framework. :frowning:

Key features are the ability to dynamically set the server, share files easily between systems, and distribute work load by calling SBASIC or Skewworks Apps on other devices. Devices currently processing are flagged as busy until completion. You can also create distribution lists so the server can update every single device without actually having to ping them all (each device hands off the commands, updates, etc, down the line and then reports back to the main server with success/failure data).