If you *must* use .NET System.IO.Ports.SerialPort

Microsoft .NET FrameworkAs an embedded developer who writes desktop software mostly for configuration of, and data download from, peripheral devices, I use serial data streams a lot.  Mostly USB virtual serial posts from FTDI, but also the USB Communication Device Class and real 16550-compatible UARTs on the PCI bus.  Since looking at data through an in-circuit emulator debug interface is generally a miserable experience, getting serial data communication with a custom PC application is essential to analyzing data quality and providing feedback on hardware designs.  C# and the .NET Framework provide a rapid application development that is ideal for early development that needs to track changing requirements as hardware designs evolve.  Ideal in most respects, I should say.

The System.IO.Ports.SerialPort class which ships with .NET is a glaring exception.  To put it mildly, it was designed by computer scientists operating far outside their area of core competence.  They neither understood the characteristics of serial communication, nor common use cases, and it shows.  Nor could it have been tested in any real world scenario prior to shipping, without finding flaws that litter both the documented interface and the undocumented behavior and make reliable communication using System.IO.Ports.SerialPort (henceforth IOPSP) a real nightmare.  (Plenty of evidence on StackOverflow attests to this, from devices that work in Hyperterminal but not .NET because IOPSP makes setting certain parameters mandatory, although they aren’t applicable to virtual ports, and closes the port on failure.  There’s no way to bypass or ignore failure of these settings during IOPSP initialization.)

What’s even more astonishing is that this level of failure occurred when the underlying kernel32.dll APIs are immensely better (I’ve used the WinAPI before working with .NET, and still do when I want to use a function that .NET doesn’t have a wrapper for, which notably includes device enumeration).  The .NET engineers not only failed to devise a reasonable interface, they chose to disregard the WinAPI design which was very mature, nor did they learn from two decades of kernel team experience with serial ports.

A future series of posts will present the design and implementation of a rational serial port interface built upon, and preserving the style of, the WinAPI serial port functions.  It fits seamlessly into the .NET event dispatch model, and multiple coworkers have expressed that it’s exactly how they want a serial-port class to work.  But I realize that external circumstances sometimes prohibit using a C++/CLI mixed-mode assembly.  The C++/CLI solution is incompatible with:

  • Partial trust (not really a factor, since IOPSP’s Open method also demands UnmanagedCode permission)
  • Single-executable deployment (there may be workarounds involving ILMerge or using netmodules to link the C# code into the C++/CLI assembly)
  • Development policies that prohibit third-party projects
  • .NET Compact Framework (no support for mixed-mode assemblies)

The public license (as yet undetermined) might also present a problem for some users.

Or maybe you are responsible for improving IOPSP code that is already written, and the project decision-maker isn’t ready to switch horses.  (This is not a good decision, the headaches IOPSP will cause in future maintenance far outweigh the effort of switching, and you’ll end up switching in the end to get around the unfixable bugs.)

So, if you fall into one of these categories and using the Base Class Library is mandatory, you don’t have to suffer the worst of the nightmare.  There are some parts of IOPSP that are a lot less broken that the others, but that you’ll never find in MSDN samples.  (Unsurprisingly, these correspond to where the .NET wrapper is thinnest.)  That isn’t to say that all the bugs can be worked around, but if you’re lucky enough to have hardware that doesn’t trigger them, you can get IOPSP to work reliably in limited ways that cover most usage.

I planned to start with some guidance on how to recognize broken IOPSP code that needs to be reworked, and thought of giving you a list of members that should not be used, ever.  But that list would be several pages long, so instead I’ll list just the most egregious ones and also the ones that are safe.

The worst offending System.IO.Ports.SerialPort members, ones that not only should not be used but are signs of a deep code smell and the need to rearchitect all IOPSP usage:

  • The DataReceived event (100% redundant, also completely unreliable)
  • The BytesToRead property (completely unreliable)
  • The Read, ReadExisting, ReadLine methods (handle errors completely wrong, and are synchronous)
  • The PinChanged event (delivered out of order with respect to every interesting thing you might want to know about it)

Members that are safe to use:

  • The mode properties: BaudRate, DataBits, Parity, StopBits, but only before opening the port. And only for standard baud rates.
  • Hardware handshaking control: the Handshake property
  • Port selection: constructors, PortName property, Open method, IsOpen property, GetPortNames method

And the one member that no one uses because MSDN gives no example, but is absolutely essential to your sanity:

  • The BaseStream property

The only serial port read approaches that work correctly are accessed via BaseStream.  Its implementation, the System.IO.Ports.SerialStream class (which has internal visibility; you can only use it via Stream virtual methods) is also home to the few lines of code which I wouldn’t choose to rewrite.

Finally, some code.

Here’s the (wrong) way the examples show to receive data:

port.DataReceived += port_DataReceived;

// (later, in DataReceived event)
try {
    byte[] buffer = new byte[port.BytesToRead];
    port.Read(buffer, 0, buffer.Length);
    raiseAppSerialDataEvent(buffer);
}
catch (IOException exc) {
    handleAppSerialError(exc);
}

Here’s the right approach, which matches the way the underlying Win32 API is intended to be used:

byte[] buffer = new byte[blockLimit];
Action kickoffRead = delegate {
    port.BaseStream.BeginRead(buffer, 0, buffer.Length, delegate (IAsyncResult ar) {
        try {
            int actualLength = port.BaseStream.EndRead(ar);
            byte[] received = new byte[actualLength];
            Buffer.BlockCopy(buffer, 0, received, 0, actualLength);
            raiseAppSerialDataEvent(received);
        }
        catch (IOException exc) {
            handleAppSerialError(exc);
        }
        kickoffRead();
    }, null);
}
kickoffRead();

It looks like a little bit more, and more complex code, but it results in far fewer p/invoke calls, and doesn’t suffer from the unreliability of the BytesToRead property.  (Yes, the BytesToRead version can be adjusted to handle partial reads and bytes that arrive between inspecting BytesToRead and calling Read, but those are only the most obvious problems.)

Starting in .NET 4.5, you can instead call ReadAsync on the BaseStream object, which calls BeginRead and EndRead internally.

Calling the Win32 API directly we would be able to streamline this even more, for example by reusing a kernel event handle instead of creating a new one for each block.  We’ll look at that issue and many more in future posts exploring the C++/CLI replacement.

56 Comments

56 Responses to If you *must* use .NET System.IO.Ports.SerialPort

  1. Bob Hourigan says:

    Couldn’t agree with you more about the System.IO.Ports functionality. I tried using it with some success then discovered little quirky problems that you can’t put your finger on, i.e. extreme overrun when using Xon/Xoff handshake. I tried using the built in handshake functions and my own to no avail. Also noted that the reading of the hardware I/O input pins was also unreliable. I am now using the Marshall software libraries that uses the API. Their interface with the Visual Studio environment isn’t great but I have working reliable Comms and will fix the interface when time permits.

    • Ben Voigt says:

      Bob, are you using a publicly available software library for wrapping the Win32 serial port API? Or did you mean P/invoke and Microsoft’s System.Runtime.InteropServices.Marshal class?

  2. Kris Janssen says:

    Dear Ben,

    I eventually stumbled on this post by reading many of your answers on StackOverflow. Clearly you know COM :)

    Until now I have often used a particular implementation of IOPSP that is available in the “Termie” project on CodeProject (http://www.codeproject.com/Articles/23656/Termie-A-Simple-RS-Terminal).

    But as it also relies on Read() I have indeed found it to be quite unreliable at times.

    Rolling my own WinAPI wrapper would probably take me too long given my limited experience so I am quite interested to learn more on your take on this approach!

  3. Kris Janssen says:

    Bob might be referring to a commercial library indeed (http://www.marshallsoft.com)

  4. Jon Yutkowitz says:

    Ben, I am currently in the process of porting a VB6 serial port app to .NET and have found the performance to be inadequate. The app sends a series of messages with responses, before posting the next message. The intermessage gap is several milliseconds larger for the .NET version.

  5. Jon Yutkowitz says:

    My app is using the DataReceived event, as well as Read(), Write(), and BytesToRead. I am curious if I will see better performance using BaseStream ReadAsync() but not exactly clear on how to handle new data arriving as efficiently as possible.

    • Ben Voigt says:

      Reading from the BaseStream will get the data to your application with low overhead. But that’s generally a small part of the processing cost; ultimately how efficiently your serial processing is depends on your code that buffers, packetizes, and parses it.

  6. WB says:

    Could you expand upon line 8 in the recommended approach? Should it work like the IOSPS DataReceived event or something else entirely?

    • Ben Voigt says:

      At a high level it is a similar concept to the DataReceived event because it allows the application to respond to incoming data in an event-driven fashion. Important differences are that the event actually carries the received data with it, so the application is all set for processing serial data, and the implementation does this with a single API call, which is far lower overhead.

  7. WB says:

    Could you add more context? I get a compile error on line 13: use of unassigned local variable. I’m using VS 2010 with .Net 4.0.

    BTW, in first comment, IOSPS should be IOPSP.

    • Ben Voigt says:

      That looks like a limitation in the compiler’s dataflow analysis; the variable is definitely assigned before use. Initialization by itself isn’t adequate, if the initializer expression passes the variable by reference to another function. But this code doesn’t, it only constructs a delegate using the variable; it doesn’t actually execute it.

      In any case you can work around it by splitting up initialization and delegate creation.

      Action kickoffRead = null;
      kickoffRead = delegate {

  8. Marya says:

    Thanks Ben for all the explanation. Do you have any sample code on how to use “ReadAsync” ? I do not seem to find a good document on how to use it.

    Thanks

  9. dominikjeske says:

    Can You give full example on sending a message and receiving a response using BaseStream. I have tried this but it is not working for me? Sample would be very helpful

    • Marya says:

      This might be useful:

      for writing, you can use
      await sp.BaseStream.WriteAsync(buffer ,0, buffer.Length);

      for reading,
      byte[] buffer = new byte [7];
      int read = 0;
      while (read < 7)
      read += await sp.BaseStream.ReadAsync(buffer , read, 7 – read);

      these commands should be used within a function that uses "async" as a modifier. For example:
      private async void DoSomething()
      {
      // Read or write like explained earlier
      }

    • dominikjeske says:

      Unfortunately I have to use .NET 4.0. I’m looking for some full example from send to receive an answer. I have sent some data but I have timeout when I’m waiting for response.

    • Ben Voigt says:

      If you ask a question on StackOverflow you are welcome to bring it to my attention here. But the blog platform is not designed for Q&A. Besides, at SO you’ll get input from multiple experts. To address your particular case, appropriate timeouts are specific to the device you’re communicating with; I could give sample code that works perfectly in my environment and you could still get timeouts.

  10. Marya says:

    Is there any way the we read from the buffer without knowing how long is the buffer? Thanks

    • Ben Voigt says:

      Sure, if you set a read timeout of zero and issue a read request, you’ll get as much data transferred to your buffer as is already received (up to the limit of the size you made your buffer). This is a much better approach than querying the buffer size. In general, just getting the data is more efficient and involves fewer race conditions than asking questions about the data and then retrieving it.

  11. Marya says:

    As far as I know, readAsync or WriteAsync does not support Readtimeout or Writetimeout. Even creating a manual timeout and setting a condition would not work since await operator is sitting there to get more data, although reading task is complete.

    • Ben Voigt says:

      “As far as I know” leads to false assumptions. You can avoid bad assumptions by either testing, or inspecting the implementation logic (JetBrains dotPeek and Red Gate .NET Reflector are good options here). It seems like you did neither.

      There are no serial port functions in the Win32 API that don’t respect timeout. The only way the .NET wrapper would not finish on timeout is if it detected a partial read and looped to read the rest. But I used dotPeek to verify that there is no such logic.

      Stream’s ReadAsync is just a wrapper around BeginRead/EndRead, and that returns once a Win32 ReadFile operation completes. There’s no loop to fill the entire buffer. A timeout will cause the ReadFile operation to complete, which means SerialStream.EndRead is called, and Stream.ReadAsync’s task completes also.

      Now, it is true that SerialStream.ReadTimeout is not as flexible as Win32 SetCommTimeouts. But it does support the desired combination “A (ReadIntervalTimeout) value of MAXDWORD, combined with zero values for both the ReadTotalTimeoutConstant and ReadTotalTimeoutMultiplier members, specifies that the read operation is to return immediately with the bytes that have already been received, even if no bytes have been received.” You get that from setting SerialStream.ReadTimeout=0.

  12. Alex says:

    hi Ben, do you know how to properly use SerialPort using .NET compact framework? MSDN said that “The .NET Compact Framework does not support the asynchronous model with base streams.”

  13. Matt says:

    I’m getting compile error due to kickoffRead(); being an unassigned local variable in VS2012. Any thoughts?

  14. Thanks for this posting! Is there anything special you want to note about writing to a serial port? What I’ve gleaned so far is that writing is blocking and should be done on a separate thread.

    • I should add that I’m working with .NET 4.0 & C#.

    • Ben Voigt says:

      Writes don’t need a separate thread, because they don’t block as long as there is room in the kernel buffer. If there isn’t room in the kernel buffer, it means you’re writing large amounts of data at a higher rate than the port can drain it — but you configured the port baud rate. (Exception: when the port is only emulating asynchronous serial and baud rate is meaningless) In practice usually writes are interspersed with reading responses, and the write buffer never overflows. Finally, if you’re worried about overfilling the buffer, use WriteAsync rather than using a separate thread.

  15. Jason A says:

    You originally wrote “A future series of posts will present the design and implementation of a rational serial port interface built upon, and preserving the style of, the WinAPI serial port functions.”

    Are you still going to post those details and implementation ? I’m sure there are plenty of people out there (like me !) who would love to have a more robust alternative to the one Microsoft left us with …

    • Ben Voigt says:

      Jason, yes those are still planned, but my dissertation committee asked for some edits before publication so those are a higher priority for my writing bandwidth this month. More serial port blog posts in March. Probably.

  16. FerZ Franco says:

    great code, I’m just having a problem, when I close the port: serial.close() I recieve an System.InvalidOperationException, “The BaseStream is only available when the port is open” dont know how to close it, I try catch everything and the problem persists, please help me!!! :), thanks

    • Ben Voigt says:

      That’s not nearly enough information to understand what’s going on. I suggest you ask a question on StackOverflow, providing all the relevant details including some code snippets and the order of calls. Cancellation of overlapped or async operations may play into your issue as well.

  17. Postmaster says:

    Awesome code, one thing dude, I’m having trouble closing the port using the private void Window_Closing event,

    it throws: System.InvalidOperationException, “The BaseStream is only available when the port is open”. How did you managed to close the stream / port ? thanks

  18. Tony says:

    Are the issues described above dependent on the type of serial device (legacy RS232 vs USB)? I know the former can be a bit more complicated (3-wire, 5-wire, 8/9 wire, hardware handshaking, etc.) than the simplified interface that USB provides.

    • Ben Voigt says:

      Some of the issues are related to the fact that the software APIs support all the handshaking modes, and many USB virtual serial drivers don’t. (Some do, actually USB is far more complicated than RS-232 even with all the toppings. Virtual COM port drivers attempt to hide that complexity… but often the differences surface to the application, and IOPSP doesn’t handle missing features very well.)

  19. Jupiter says:

    What are Stream replacement for SerialPort’s DiscardInBuffer/DiscardOutBuffer ?

    • Ben Voigt says:

      Interesting point. SerialStream (the class implementing BaseStream) has these, but they are inaccessible.

      In the end though, it is a moot point. The SerialPort class appears to make these operations available, but actually makes them useless. (Because events are handled on thread pool threads, there’s a race condition, making the effect of these functions completely unpredictable.)

      The underlying Win32 function PurgeComm is not something you’d ever use while you have background operations in progress. Correct usage is to cancel your background operations, empty the buffers, and start them up again. Which unfortunately you can’t do with the SerialPort class since it always has that WaitCommEvent background operation for event generation, and you can’t do with SerialStream because they aren’t public. So throwing away the wrapper classes and using the Win32 API (possibly through a better wrapper) looks like the only option, if your application really really needs these operations.

  20. Chris says:

    Hi Ben,
    Great post.

    However, I’am trying to implement this code on a WinCE7/ARM with .NETCF 3.5.

    I’am getting the NotSupportedException as described here when I run my app:
    https://msdn.microsoft.com/en-us/library/system.io.ports.serialport.basestream%28v=vs.90%29.aspx

    Am I missing something?

    Thx,

  21. Andreas D says:

    I’m pretty much “suffering” with SerialPort as well, especially the fact it deals with the device “suddenly disappearing” while the port is opened…

    A question regarding your code: If I were to use ReadAsync, I would call .ContinueWith(kickoffRead) on the returned Task object to achieve the same as passing kickoffRead as AsyncCallback to BeginRead?

    Also, what do I need to consider when closing the port? Maybe passing a CancellationToken to the ReadAsync method?

  22. Daniel says:

    Hi Ben,

    Thanks for your profound explanations!

    Our app suffers from sporadic RXOver errors and I suspect the combined use of properties DataReceived and BytesToRead.

    However, I cannot use your approach as we are on Windows CE and Microsoft states in a laconic comment that “The .NET Compact Framework does not support the asynchronous model with base streams.”

    What could be a solution for Windows CE / Compact Framework?

    • Ben Voigt says:

      The solution I’ll be presenting is direct use of the Win32 API Communication Functions (which Windows CE does support). Unfortunately, the mixed-mode C++/CLI assemblies I’ll be using to make C# programmers’ lives easier have no equivalent on Compact Framework, so you’d need p/invoke and all the declaration translation concomitant with p/invoking a complex API. The serial port functions themselves aren’t too bad. Detection of port adapter plug and unplug events… well maybe those aren’t of as much concern on .NETCF

  23. Chris says:

    Hi Ben,

    Any news on when you might post more on Serial port implementations?

    Thanks

    • Ben Voigt says:

      My writing calendar, according to priority, is roughly thus: Get dissertation deposited with university, prepare and submit “Supplemental Experience Record” to the Texas Board of Professional Engineers so I can sit for the exam in the fall, continue writing blog entries about serial ports in general, using them under Win32, and the EventDrivenSerial C++/CLI library that I’ve written to supplant System.IO.Ports.SerialPort. So, maybe in a month, and you certainly should see more of my posts this summer.

  24. Neville says:

    Hi Ben,

    Would it be possible to somehow get a beta version of your C++/CLI library?

    We could *really* use it on our current project where we are having performance issues (CPU loading mostly) with the standard C# serial port.

    Thanks!

  25. ben says:

    I’ve been researching the use of serial libraries as it pertains to calling from python.

    I’ve had trouble calling dll’s that are written with .net (c# etc). I’m not sure if it’s a managed vs unmanaged thing, but the dlls that are written in C/C++ exports the function names that have no problems being called within python (windows). It looks like I may not be alone:
    https://www.nuget.org/packages/UnmanagedExports

    Any thoughts on this subject? Is there a way to write a windows library (.net or otherwise) that does NOT use System.IO?

    • Ben Voigt says:

      Calling from one framework into a serial wrapper library of another framework seems like a bad idea, even if the wrapper weren’t as poorly designed as IOPSP.

      System.IO is not the OS API. It would be very rare to use it from any environment except .NET.

      I suggest either using python’s Win32 API support to call the OS APIs directly or an existing python serial library.

  26. Andy says:

    I would like to send some data to the serial device then read the data that it replies with, but for one reason or another I’m not currently able to utilise the async methods or DataReceived event.

    My current implementation is simply a loop that repeatedly calls the Read() method until I’ve received the entire response. It “works”, but I often get a number of TimeoutExceptions (which I just swallow). I assume these are due to the device not replying straight away and/or the reply (which can be many kb) being sent piecemeal.

    Is this an acceptable approach or can you offer any advice for what I’m doing?

    I did wonder if I could avoid the timeouts by waiting until there is actually data at the port (e.g. “while (BytesToRead == 0″) before attempting a Read(), but your article seems to imply that BytesToRead can be problematic?

    • Ben Voigt says:

      Busy-waiting is a terrible approach; you shouldn’t do it even if BytesToRead weren’t broken.

      It sounds like your problems arise from the “for some reason or another”. If you’re ignoring timeout exceptions, that means you have set the timeout too short — change it. Use the serial port the way it was designed.

  27. david says:

    Is it possible to invoke java code from .net based application. If yes, take a look at this new open source library github.com/RishiGupta12/serial-communication-manager seems like it is promising.

    • Ben Voigt says:

      This post specifically addresses the case where developers are locked into the framework SerialPort class. When using other libraries is allowed, there are much better options than paying the .NET Java integration cost.

  28. rx says:

    Why not publish your code “as is” on github?

    • Ben Voigt says:

      One does not simply upload code without scrubbing all mention of customer-specific requirements and defect reports from the comments. Not even under an “as-is” license.

  29. Gerhart says:

    It’s a good idea to have serial port class in the framework, but I use a 3rd party class from Zyl Soft, which is more stable. There are other 3rd party components as well.

  30. mwpowellhtx says:

    But for configuration issues, which I gather could well be routed through a C++/CLI vernier, which would require, at some level, creating a reader/writer/session using the CreateFile API… Apart from that aspect, everything else could be done through the FileStream, BaseStream, elements? Probably am forgetting bits about it.

    • Ben Voigt says:

      You’d think that, but as far as being a wrapper for the underlying API goes, BaseStream isn’t anywhere near complete. Dmitry just ran into another example.

  31. Thank you for the article! It was really great to find it. And I really appreciate your explanation.

    I have several questions about catching Frame error with this approach.

    Context: I have 1Mb one directional stream of data from a device to PC. Chunks are copied for several times, so it is possible to recover if one is lost. Frame error happens from time to time. Frame error usually means that some data is physically lost. But for me it is essential to get as much possible around it.

    ErrorReceived and DataReceived showed complete incompetence, as it stated in your article.

    Using BaseStream approach, at:
    catch (IOException exc) {
    handleAppSerialError(exc);
    }

    Does Frame error show up there?

    Is it synchronous, I mean, do all bytes up to byte when Frame error occurred are received?

    If I continue and call BeginRead after this, would I receive data from the first byte port recovered from Frame error?

    P.S. I can move it to stackoverflow. As it is convenient for you.

    • Ben Voigt says:

      Well, ClearCommError is the underlying API for detecting framing errors, and using dotPeek I see that it’s called in three places — the BytesToRead and BytesToWrite properties and the infrastructure for raising the ErrorReceived event on the threadpool. BytesToRead and BytesToWrite clear the status without reading it — obviously a very bad thing if you use them. As far as BaseStream.ReadAsync (or BeginRead) is concerned, you won’t get an exception if you encounter a framing error (which is probably a good thing, since you DID want to get the bytes that were successfully received).

      The really bad thing is that I can’t see anything at all that sets DCB flag bit 14 (fAbortOnError), so you get whatever setting you inherit from the last software that used the port.

      Feel free to ask on StackOverflow, but at this point my diagnosis is that to discover framing errors cleanly, you’ll need to use the Win32 API, either using p/invoke or C++/CLI. If you’re ok with just reading past them, the BaseStream read functions probably do that.

Add your thoughts