Main Site Documentation

Is it totally impossible to serialize a struct or class to byte array somewhat dynamically?


#1

sigh
Unsafe, the Marshal Class, Custom attributes, Generics. all of them is missing from the micro framework. im trying to find a dynamic way of converting structs or classes to byte arrays.
System.Runtime.InteropServices.Marshal would be perfect. doesnt work,
next on list, i made a custom attribute class


class attribtest{
 [MarshalPos  (0)]
 public   UInt16 X;
 [MarshalPos (2)]
 public   UInt16 Y;
 [MarshalPos (4)]
 public UInt16 Z;
}

[MarshalPos] is a custom attribute. it compiles. but there is no way in netMF to read the attributes, bummer!

ok. third, enumerate all fields. this works. i can get their type. but not size. i would have to parse fieldtype.

today is not a good day being me. enumerating all fields makes me have to write a serializer/deserializer unless i want to enumerate all fields in the get/set accessor in each field. which would most certainly not be very fast and bloat the code.

Has anyone of you made something like this before and have a really smart solution?


#2

Use insertvalueintoarray method in utilities class. It is faster and more suited for embedded devices.

I think this is explained in the serialization tutorial. Have you checked it?


#3

MarshallByRefObject is not missing nor is Serializable


#4

Maybe he is using panda


#5

I am using a Domino, i’ll look into those things you’ve mentioned. Thanks! Im not sure which way to go, as you might recall im working on getting kinect to work on netMF, i’ve decided to port all of libfreenect to managed code and there are alot of structs and pointers, memcpy and so on which is rather tricky to port since i have to make byte arrays and copy structures/class objects into. well i just want to find the best solution. And if it wasnt for GHI’s hard work this wouldnt even be possible in the ways i rant about :wink:


#6

Serialize, don’t marshal otherwise events will be a beast. Unless your not raising events, then marshal instead.


#7

Reflection.Serialize throws a notimplementedexception

even if it was working on domino, does it not add lots of header and field name like the binaryserializer in the “full” framework?

InsertValueIntoArray was close to what im looking for. though i think GHI should have overloads that match the one in Microsoft.SPOT.Hardware.Utility for cleaner looking code!

but still if i have a struct


struct somestruct {
UInt16 X;
UInt32 Y;
}

As far as i know, there is no simple way to make a four byte array from it. so im writing a helper class for it,
At the end of the day it comes down to balance. i want these methods so i dont have to repeat myself and having to count byte positions in arrays and doing shifts x times into byte in every method. but the result is more and slower code to fit on the Domino.
32gig memory, 8 CPU cores, loads of diskspace and gigabit ethernet is bad influence :wink:


#8

Serializing [italic]does[/italic] add more headers but is pretty much the only way to throw events through AppDomains in NETMF because of other missing things. The extra overhead is why I said to only use it if you need to put events through the domain :wink:


#9

Data serialization infers data deserialization.
The biggest problem with dynamic serialization is the assumptions that the worker function operates under regarding:

  • endianness of the source and destiation data
  • string encodings (NetMF only supports UTF8)
  • determining the data structure from the byte stream (for deserialization; can’t have one without the oither, yano?).

I really don’t see how you could build a “generic (de)serializer” without imposing some level of structural knowledge on the worker components.
…but then, I’m no CS…

Just for S&G, I built http://code.tinyclr.com/project/402/gadgeteer-wpad-tester/ to examine the process of data (de)serialzation on NetMF.
Probably not the most memory-efficient code (spoiled by Windows coding habits), but it works…


#10

How many distinct structs do you have to serialize?

If it isn’t too many. Why not put the metadata in the struct, like this (excuse the pseudo code and the phone formatting)…

[Code]
Struct somestruct
{
Byte TypeId = MyTypeEnum.U16U32;
Uint16 x;
Uint32 y;
}



Then pick the serializer and deserializer by looking at the TypeId field in a switch statement.

Yes, this wastes one byte per instance but it carries the meta data you're looking for.

#11

that would only work if you make your TypeId static as follow:


        struct somestruct
        {
            static Byte TypeId =  MyTypeEnum.U16U32;
            UInt16 x;
            UInt32 y;
        }


#12

[quote]How many distinct structs do you have to serialize?

If it isn’t too many. Why not put the metadata in the struct, like this (excuse the pseudo code and the phone formatting)… [/quote]

While that might work for simple structs as we’re “exampling”, it becomes very nasty for complex structs such as those seen in network traffic.

As long as the communication is fairly simple (and small), I can see a generic library, but not for complex tasks such as app-layer protocol (de)constructing.

…of course, I’m willing to be wrong (cuz I lern stuf)… :wink:


#13

Ok, I see where you’re going with this.

First, to create a generic parser to convert binary wire data to structured classes you need some kind of mapper and a parser.

With serialization like XML and the lighter JSON, the schema is defined with the data so all you need is a parser.

With high speed “raw” protocols the structure is defined in a standard and your parser no longer is generic but specific.

You should not be decorating your classes to describe specific structures if you want to create a generic parser. Does that make sense?

The next problem is that NETMF does not give you control over the position of variables in memory. This means you cannot use “binary packing” to deserialize and that means you’ll have to use an interpreter. That will be slow and defeats the purpose of dealing with high speed binary structures.

So, much as it pains me, I’d have to say do this kind of thing in RLP. There you have access to the metal and can happily hardcode the low level binary work. Put a class over a known RLP array to represent the data as .Net objects.


#14

Completely agree, but sometimes you just gotta do things the hard way to understand the underlying mechanisms…
Luckily, I started with something I already now - packet parsing. That made it a lot easier to work through the problem in NetMF. I’m just thankful for the MS/GHI utility additions - that made things a bit simpler.

RLP - that’s next on the learning ladder.
I need to get my head around the differences in NetMF first :-p