NMEA parser without strings

Instead of a native-code NMEA parser, you could generalize this into something like regex or sscanf and make it a generalized parser - e.g., template and struct in; populated struct out. What makes it an NMEA are the sentence templates. That could be written in managed or native code.

I would say ‘just use regex’ but you would still end up with some memory allocations since regex first creates an intermediate string that has to be converted to a numeric. But in the end, a generalized scanning grammar that includes the ability to convert numerics seems like a more reusable solution.

You are right. Regexp is very memory intensive and slow. At least with the managed implementation. And you end up with strings anyway.
To me, it’s not a good idea to use it for parsing NMEA data at the rate and quantity it comes from the GPS.

Before coding this byte parser, I’ve tried this code :

>     var receivedString = "$GPRMC,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,003.1,W*6A";
> var expRMC = "([$]GPRMC)[,]([0-9]{6})[,]([AV]{1})[,](.*)[,]([NS]{1})[,](.*)[,]([EW]{1})[,](.*)[,](.*)[,](.*)[,](.*)[,]([EW]{1})([*][0-9a-f]{2})";
>                 var NMEAPattern = "[$]([A-Z]{2})([A-Z]{3}).*[*]([0-9a-f]{2})";
> 
>                 Regex validNMEA = new Regex(NMEAPattern, RegexOptions.IgnoreCase);
> 
>                 if (validNMEA.IsMatch(receivedString))
>                 {
>                     Match m = validNMEA.Match(receivedString);
>                     switch (m.Groups[2].Value)
>                     {
>                         case "RMC":
>                             Regex rRMC = new Regex(expRMC, RegexOptions.IgnoreCase);
>                             Match mRMC = rRMC.Match(receivedString);
>                             RMCSentence.FixTime = mRMC.Groups[2].Value != String.Empty
>                             ? new TimeSpan(Convert.ToInt32(mRMC.Groups[2].Value.Substring(0, 2)), Convert.ToInt32(mRMC.Groups[2].Value.Substring(2, 2)), Convert.ToInt32(mRMC.Groups[2].Value.Substring(4, 2)))
>                             : new TimeSpan(0);
>                             RMCSentence.Status = String.IsNullOrEmpty(mRMC.Groups[3].Value) ? Char.MinValue : mRMC.Groups[3].Value[0];
>                             RMCSentence.Latitude = (Single)Double.Parse(mRMC.Groups[4].Value) / 100;
>                             RMCSentence.LatitudeHemisphere = String.IsNullOrEmpty(mRMC.Groups[5].Value) ? Char.MinValue : mRMC.Groups[5].Value[0];
>                             RMCSentence.Longitude = (Single)Double.Parse(mRMC.Groups[6].Value) / 100;
>                             RMCSentence.LongitudePosition = String.IsNullOrEmpty(mRMC.Groups[7].Value) ? Char.MinValue : mRMC.Groups[7].Value[0];
>                             RMCSentence.SpeedKnots = (Single)Double.Parse(mRMC.Groups[8].Value);
>                             RMCSentence.SpeedKm = RMCSentence.SpeedKnots * 1.852f;
>                             RMCSentence.TrackAngle = (Single)Double.Parse(mRMC.Groups[9].Value);
>                             RMCSentence.MagneticVariation = (Single)Double.Parse(mRMC.Groups[11].Value);
>                             RMCSentence.MagneticVariationDirection = String.IsNullOrEmpty(mRMC.Groups[12].Value) ? Char.MinValue : mRMC.Groups[12].Value[0];
>                             RMCSentence.Checksum = (Byte)Convert.ToInt32(mRMC.Groups[13].Value.Substring(1, 2), 16);
>                             break;
>                     }
>                 }

This does indeed work. But unfortunately it’s not efficient at all.

But my point is that you can still get the memory win without making this an NMEA-specific parser. Just struct+template in and populated struct out, in either native or managed code (though admittedly, without reflection, the native code implementation is more complex).

It becomes an NMEA parser when you use NMEA templates as the input.

My idea is to implement a helper, nothing specific to GPS. Like CSV parser. Which then we can use for this example and other things, like simple configuration values (network config from a text file for example).

1 Like

Config files are a whole 'nother basket of worms, but yes, in general, I agree.

For config files, I have been using bson because it is the most compact storage format that can be directly read (no decompression) while still allowing for schema and version management and can be read in place (without copying elements). Of course, if config files have to be human readable, then text or json is better, but also less space efficient.

I’ve also moved to logging via bson, and logging just the message type and data payload and then transforming that to readable text only when it needs to be human readable (and generally, that’s on a server or desktop machine).

1 Like

I have finally fixed all issues I had so far. Those were mainly in GSV sentences.

Now the parser can handle such GSV sentences :

$GPGSV,3,1,11,03,03,111,00,04,15,270,00,06,01,010,00,13,06,292,00*74
$GPGSV,3,3,11,22,42,067,42,24,14,311,43,27,05,244,00,,,,*4D
$GPGSV,4,4,13,30,31,066,28*40
$GLGSV,3,3,10,84,06,036,,,,,18*52
$GLGSV,1,1,02,72,,,29,74,,,19*62
$GLGSV,1,1,02,65,50,140,28,,,,32*5F
$GPGSV,4,4,14,36,29,144,31,49,35,178,*74
$GPGSV,1,1,00*79

There are some sentences with “inconsistent data”, others with partial data and others with missing expected data…
Many of the sentences above have been received by real hardware (GNSS Click, GNSS 4 Click and GNSS Zoe Click), so I’m pretty confident in the parser.

FYI, parsing 16.000 sentences (1.000 * 16) is using only 19KB of memory. Again, this is a stable consumption. No matter if you parse 1, 10, 100 or 10.000 sentences.

Before loop :
+-----------+------------+------------+
| Memory    | Used       | Free       |
+-----------+------------+------------+
| Managed   |    173,968 |    333,824 |
| Unmanaged |          0 | 33,554,404 |
+-----------+------------+------------+

After loop :
+-----------+------------+------------+
| Memory    | Used       | Free       |
+-----------+------------+------------+
| Managed   |    193,008 |    314,784 |
| Unmanaged |          0 | 33,554,404 |
+-----------+------------+------------+

Parser is available now on our Github repo.
I have reactivated a PR on TinyCLR/Drivers repository.

Edit : Here is the program I’ve used for memory stress tests.

class Program
    {
        private static readonly String GGAString1 = "$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47";
        private static readonly String GGAString2 = "$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,0123*47";
        private static readonly String GSAString = "$GLGSA,A,3,04,05,,09,12,,,24,,,,,2.5,1.3,2.1*39";
        private static readonly String GSVString1 = "$GPGSV,3,1,11,03,03,111,00,04,15,270,00,06,01,010,00,13,06,292,00*74";
        private static readonly String GSVString2 = "$GPGSV,3,3,11,22,42,067,42,24,14,311,43,27,05,244,00,,,,*4D";
        private static readonly String GSVString3 = "$GPGSV,4,4,13,30,31,066,28*40";
        private static readonly String GSVString4 = "$GLGSV,3,3,10,84,06,036,,,,,18*52";
        private static readonly String GSVString5 = "$GLGSV,1,1,02,72,,,29,74,,,19*62";
        private static readonly String GSVString6 = "$GLGSV,1,1,02,65,50,140,28,,,,32*5F";
        private static readonly String GSVString7 = "$GPGSV,4,4,14,36,29,144,31,49,35,178,*74";
        private static readonly String GSVString8 = "$GPGSV,1,1,00*79";
        private static readonly String RMCString = "$GBRMC,221030,A,4807.038,N,01131.000,E,022.4,084.4,101120,003.1,W*6A";
        private static readonly String VTGString = "$INVTG,220.86,T,,M,2.550,N,4.724,K,A*34";
        private static readonly String HDTString = "$GAHDT,274.07,T*03";
        private static readonly String GLLString = "$GNGLL,4404.14012,N,12118.85993,W,001037.00,A,A*67";
        private static readonly String UNKString = "$GNUNK,4404.14012,N,12118.85993,W,001037.00,A,A*67";

        static void Main()
        {
            TestByteArray();

            Thread.Sleep(Timeout.Infinite);
        }

        static void TestByteArray()
        {
            Byte[] GGA1 = Encoding.UTF8.GetBytes(GGAString1);
            Byte[] GGA2 = Encoding.UTF8.GetBytes(GGAString2);
            Byte[] GSA = Encoding.UTF8.GetBytes(GSAString);
            Byte[] RMC = Encoding.UTF8.GetBytes(RMCString);
            Byte[] GSV1 = Encoding.UTF8.GetBytes(GSVString1);
            Byte[] GSV2 = Encoding.UTF8.GetBytes(GSVString2);
            Byte[] GSV3 = Encoding.UTF8.GetBytes(GSVString3);
            Byte[] GSV4 = Encoding.UTF8.GetBytes(GSVString4);
            Byte[] GSV5 = Encoding.UTF8.GetBytes(GSVString5);
            Byte[] GSV6 = Encoding.UTF8.GetBytes(GSVString6);
            Byte[] GSV7 = Encoding.UTF8.GetBytes(GSVString7);
            Byte[] GSV8 = Encoding.UTF8.GetBytes(GSVString8);
            Byte[] VTG = Encoding.UTF8.GetBytes(VTGString);
            Byte[] HDT = Encoding.UTF8.GetBytes(HDTString);
            Byte[] GLL = Encoding.UTF8.GetBytes(GLLString);
            Byte[] UNK = Encoding.UTF8.GetBytes(UNKString);

            Info();

            for (var i = 0; i < 1000; i++)
            {
                NMEAParser.Parse(GGA1);
                NMEAParser.Parse(GGA2);
                NMEAParser.Parse(GSA);
                NMEAParser.Parse(RMC);
                NMEAParser.Parse(GSV1);
                NMEAParser.Parse(GSV2);
                NMEAParser.Parse(GSV3);
                NMEAParser.Parse(GSV4);
                NMEAParser.Parse(GSV5);
                NMEAParser.Parse(GSV6);
                NMEAParser.Parse(GSV7);
                NMEAParser.Parse(GSV8);
                NMEAParser.Parse(VTG);
                NMEAParser.Parse(HDT);
                NMEAParser.Parse(GLL);
                NMEAParser.Parse(UNK);

                Thread.Sleep(20);
            }

            Info();
        }

        private static void Info()
        {
            Debug.WriteLine($"+-----------+------------+------------+");
            Debug.WriteLine($"| Memory    | Used       | Free       |");
            Debug.WriteLine($"+-----------+------------+------------+");
            Debug.WriteLine($"| Managed   | {Memory.ManagedMemory.UsedBytes,10:N0} | {Memory.ManagedMemory.FreeBytes,10:N0} |");
            Debug.WriteLine($"| Unmanaged | {Memory.UnmanagedMemory.UsedBytes,10:N0} | {Memory.UnmanagedMemory.FreeBytes,10:N0} |");
            Debug.WriteLine($"+-----------+------------+------------+\r\n");
        }
}

Hopefully it will be useful to others.

6 Likes