How to split bits into separate channels

Sometimes channel data on a file consists of arrays of bits with each bit belonging logically to it's own channel. This is one of the use cases that the "BitMask" is meant to cover.

Let's say you have a file which contains eight one-bit channels. This can be read by creating eight eight-bit channels which cover the same file space and masking in a different bit for each channel. The code looks like this:

Option Explicit

Sub ReadStore(File)
 Dim ChannelGroup : Set ChannelGroup = Root.ChannelGroups.Add("ChannelGroup")

 Dim i, Block, DAChannel
 For i = 0 to 7
   Set Block = File.GetBinaryBlock()
   Set DAChannel = Block.Channels.Add("bit" & i, eByte)
   DAChannel.Formatter.BitMask = 2^i
   DAChannel.Factor = 1 / 2^i
   Call ChannelGroup.Channels.AddDirectAccessChannel(DAChannel)
 Next
End Sub

Here are a couple of points which might not be obvious:

Because you must create a channel the size of which is measured in bytes, it is only possible to read out bits which are spaced in multiples of eight using this method. Up until now I haven't seen a spacing which made this a problem. If anyone else has go ahead and let me know.

When you call GetBinaryBlock, it does not advance the file pointer. That means that calling GetBinaryBlock multiple times from the same file position results in multiple blocks which read starting at that same file position.

It's important that you use an unsigned datatype (eByte, eU16, or eU32) to create the channels. For the signed types we've implemented sign extension (first available in DIAdem 10.0). Since you are masking out all but one of the bits, two's complement means that when the last (in this case only) un-masked bit is a one, the number is negative. You probably want a channel of 0's and 1's, and not a channel of 0's and -1's.

For this example, I set the BitMask property using a decimal number. However, sometimes it is more readable to set the BitMask using a hexidecimal number. Something like 0xff00 is difficult to input into VBScript. We've made it easier by making the BitMask property also accept strings. This means you could set it to "0xFF00", and it would behave the same as if you set the BitMask property to 65280.

When channel values aren't interleaved

There are two ways in which binary channel values are commonly written: ABCABCABC... or AAA...BBB...CCC...

In the first case, the writing application alternates between each of the channels. This is a common manner of writing data which comes from a streaming application where all channels have the same sample frequency.

In the second case the writing application wrote all the values of one channel before continuing on to the second channel. Applications which perform one measurement after the other, or applications which do some caching are more likely to write this second kind of channels. As of this writing, the DataPlugins help doesn't address this second use case with an example, so I'm posting an example here.

In the DataPlugin API each binary block represents a single channel or multiple channels interleaved in the first use case above. In order to cover the second use case, create a binary block for each channel, and set its length as the number of channel values.

I devised a simple file format to illustrate this case. This format looks like this:
[file header]
[1st channel header]
[1st channel values]
[2nd channel header]
[2nd channel values]
...
[nth channel header]
[nth channel values]

The file header looks like this:
22 1-byte characters: file format marker
1 4-byte integer: number of channels in the file.

And the channel header looks like this:
10 1-byte characters: channel name
1 4-byte integer: number of values in a channel

The code to read this file starts by reading the file header, and then reads the individual channels:

Sub ReadStore(File)
 Dim ChannelCount : ChannelCount = ReadFileHeader(File)
 Dim ChannelGroup : Set ChannelGroup = Root.ChannelGroups.Add("ChannelGroup")

 Dim i : For i = 1 to ChannelCount
   Call ReadChannel(File, ChannelGroup)
 Next
End Sub

Reading the file header is fairly straightforward. First we use the file marker to reduce our chances of trying to read a file that the DataPlugin isn't made for (more detailed checks would be possible). Then we read out the number of channels in the file.

Function ReadFileHeader(File)
 If Not CheckValidity(File) Then RaiseError "Not a valid Ducks file."
 ReadFileHeader = File.GetNextBinaryValue(eI32)
End Function

Function CheckValidity(File)
 CheckValidity = False
 Dim Identity : Identity = File.GetCharacters(22)
 If Identity = "NI example binary file" Then   CheckValidity = True
End Function

Once we know how many channels there are we can loop over that number to create those channels. In the channel reader we read out the name of each channel and the number of values it contains. Then we create a block containing one channel to read out the values of each channel. We finish off by moving the file pointer to the beginning of the next channel.

Sub ReadChannel(File, ChannelGroup)
 Dim ChannelName : ChannelName = File.GetCharacters(10)
 Dim ChannelSize : ChannelSize = File.GetNextBinaryValue(eI32)

 Dim Block : Set Block = File.GetBinaryBlock()
 Block.BlockLength = ChannelSize

 Dim DAChannel : Set DAChannel = Block.Channels.Add(ChannelName, eI32)
 Call ChannelGroup.Channels.AddDirectAccessChannel(DAChannel)

 File.Position = File.Position + ChannelSize*4
End Sub

One mistake I've made and seen others make is mixing up the units for BlockLength and File.Position. BlockLength contains the number of values in each channel in the block. If there is one channel in the block with 30 values then BlockLength is 30. If there are two such channels BlockLength is still 30. File.Position is in bytes. A block with one 30 value channel of type eI32 would cover 120 bytes out of the file. A block with two such channels would cover 240 bytes.

Acquiring the channel length will be different for each file format. Sometimes it is explicit as in the above example. Sometimes it is implicit, for example, there are n equal-length channels in the file, so divide the file size by n and you have the channel length in bytes. Use the size of the type to convert this to the channel length in number of values.

How to create a waveform channel

First off, you may be wondering what a waveform channel is. To graph data you need two channels of data: one containing the x values and one containing the y values. In a lot of measurement applications, the x values can be generated with just a start value, an increment and a size. This kind of channel is called an implicit channel in DIAdem. Instead of putting these values in an implicit channel, you can also add these values as properties to the channel containing the y values. A channel with these values is called a waveform and is displayed in DIAdem like this:



The big advantage to a waveform is that starting with DIAdem 10.0, instead of having to choose two channels for a graph in VIEW, or REPORT you can choose one. This reduces the probability of error for you or your customer when creating simple reports. It also reduces the amount of special knowledge required to make sense of your data.

Making a channel into a waveform channel is also very easy. All you have to do is add a few properties. You could for example, copy the following function into your DataPlugin and call it on your channels where you would otherwise call AddImplicitChannel:

Sub MakeWaveform(Channel,Offset,Increment,XName,XUnit)
 Call Channel.Properties.Add("wf_start_offset",CDbl(Offset))
 Call Channel.Properties.Add("wf_increment",CDbl(Increment))
 Call Channel.Properties.Add("wf_xname",XName)
 Call Channel.Properties.Add("wf_xunit_string",XUnit)
 Call Channel.Properties.Add("wf_samples",CLng(1))
 Call Channel.Properties.Add("wf_time_pref","relative")
 Call Channel.Properties.Add("wf_start_time",CreateTime(0,1,1))
End Sub

For a full documentation of what each of these properties mean, I suggest you read the DIAdem help. The really interesting properties are offset (ie start value) and increment.

(As a side note: Calling CreateTime with less than the full 9 arguments is supported starting with DataPlugin API version 1.5. If you don't wish to switch to this version then you can fill out the rest of the arguments with 0's.)

How and when to use RaiseError

Imagine that you have two programs, both of which output measurement files that you want to read into DIAdem. Both formats are written to files with the extension *.dat. Otherwise the formats are different enough that it makes a lot of sense to write a separate DataPlugin for each of them. How do you make sure that DIAdem and the DataFinder use the correct DataPlugin for the correct files?

The answer is that you need to write code in each of your two DataPlugins to recogize invalid files. Most file formats make this possible in some way or another. For example, some programs write their own names, or software version numbers somewhere close to the top of the file. Some file formats start with a header with fields containing numbers and text. If the fields can't be parsed, (for example, the value of an enumeration doesn't match its known possible values, or you find text where you were expecting a number) then that is a sign that the file doesn't match your plugin.

Once you've recognized an invalid file, the proper way to indicate the error, is to call RaiseError with a text describing the problem. How detailed your description is depends on who will use your DataPlugin. If you expect your DataPlugin to read files which users tweak by hand with a text editor, then you would help your users most with a very detailed message. If your DataPlugin reads one file format which can be written in several versions and you don't support all of them, then you could try to break the possible errors down by file format version. Otherwise it is enough just to call RaiseError with a simple statement that the file contains an unsupported format.

Now you may be thinking, "My file extension is pretty unique, so I don't have to worry about another program using the same file extension". Sometimes developers mis-estimate the likelihood that another program might have the same file extension. With only three letters the chance of someone else randomly hitting your combination is actually pretty high. Here are some other good reasons for programming some format-recognition code:

• The program which writes the file may come out in a new version which supports new features for its formats. If this happens, your DataPlugin may incorrectly read the new version of this format.
• The program which writes the file could contain a bug which causes it to occasionally write corrupted files.

In these kinds of cases you would probably rather see an error message, than wonder why your data look so odd.