X-ray FuelSpray: APD Data Processing

Raw Data File Formats

Over the course of the project, the file format of the raw data has changed as our data acquisition system has advanced. There are also different acquisition systems for different detectors. The list below gives an overview of the major formats that have been used.

Experiments using Avalanche Photodiode

Apr 1999 - Aug 1999 (Experiments Bosch1 & Bosch2)
Dec 1999 - Feb 2001 (Experiments OMC1 - Delphi3)
Sep 2001 - Jul 2005 (Experiments Bosch6 - Bosch20)
Oct 2005 - Present (Experiments Bosch22 - )

Experiments using Cornell PAD (Delphi1, Bosch5, Delphi2, Bosch9, Delphi5, Bosch17, Visteon2)
Experiments using CCD camera for absorption imaging (Bosch4)
Experiments using phase contrast imaging techniques (Bosch16)

Experiments using Avalanche Photodiode

Most of our measurements have used an avalanche photodiode (APD) to measure the x-ray intensity as a function of time. The APD response is recorded using a digital oscilloscope, and then transferred to a computer to be written to disk. We have used several different brands and models of oscilloscopes, and several different pieces of software to transfer from scope to computer. This has resulted in several different file formats for APD data, these are described below.

Apr 1999 - Aug 1999

Experiments that used this file format: Bosch1, Bosch2

For these measurements, we used a Tektronix TDS ??? oscilloscope. The data was transferred from scope to a PC using the Tektronix software package called "WaveStar". The data files were written to disk in the proprietary format of this software. They were then converted to ASCII offline using WaveStar. The details of the file format are not described here.

Dec 1999 - Feb 2001

Experiments that used this file format: OMC1, Bosch3, Delphi3

For these measurements, we used a Tektronix TDS ??? oscilloscope. The data was transferred from the scope to a Sun Workstation using software running on EPICS custom-written for us by Tim Mooney of APS. The data files were written to disk in the format of an EPICS scan. They were then converted to ASCII offline using the EPICS code called "mda". The details of the file format are not described here.

Sep 2001 - Jul 2005

Experiments that used this file format: Bosch6, Bosch7, Bosch8, Delphi4, Bosch10, Bosch11, DDC1, Bosch12, Bosch13, Visteon1, DDC2, GM-ERC1, Bosch14, DDC3, Bosch15, Bosch18, GM-ERC2, Bosch19, GM-ERC3, Bosch20

For these measurements, we used a Lecroy WavePro 960 (?) oscilloscope. The data was transferred from the scope to a PC using ActiveDSOClient.exe, a Visual C++ code running on a PC custom-written for us by Tim Mooney of APS. The data was written to disk on the PC, each data file contained a complete EPICS scan, either 1D or 2D. These scans were typically done as a transverse slice through the spray at some distance from the nozzle, therefore each data file typically contained measurements at a single axial coordinate and multiple transverse coordinates. However, there is no limitation in the file format on which position coordinates could be included in the file.

Sample Raw Data File:

File=x:\scans\scan220.dat Xmotor=15.999305 Ymotor=-2.599699 data[nY=0,nX=81] wavePoints=01000000 sampleInterval=1.000000e-009 IC2=2
*** 1000000 unsigned big-endian bytes here ***
File=x:\scans\scan220.dat Xmotor=15.999305 Ymotor=-2.534933 data[nY=0,nX=81] wavePoints=01000000 sampleInterval=1.000000e-009 IC2=69269
*** 1000000 unsigned big-endian bytes here ***

Explanation of File Format:

The file is broken up into any number of sections, each section is devoted to one particular X,Y position coordinate. At each position coordinate, a single channels of data is recorded. The sections for each position coordinate are listed one after another. The file may begin with a blank line.
Each position coordinate begins with an ASCII text header. This text header contains the x and y coordinate of the measurement, the record length of the data (wavePoints=n), the time step between data points (sampleInterval=i), and several other parameters. The header consists of key=value pairs separated by spaces, and is concluded with an end-of-line character. The number as well as the order of these key value pairs is fixed, with one exception: IC2=xxx may not be present in the earlier experiments.
Following the coordinate header, the APD data is recorded. The data is recorded as big-endian unsigned bytes, the number of bytes is indicated by the value of wavePoints. The data indicates the ADC value of the the data acquisition device, there is no way to recover the actual value in volts.
Additional sections for additional coordinates follow in the order they were measured, there are typically no blank lines between coordinates.

Oct 2005 - Present

Experiments that used this file format: Bosch22, Bosch23, Bosch24, GM-ERC4

For these measurements, we used Yokogawa DL7100 and DL7480 oscilloscopes. The data was transferred from the scope and written to files on a PC using DataGrabber, which is a custom-written Java code developed by our group. Each data file contains a complete EPICS scan, either 1D or 2D. These scans are typically done as a transverse slice through the spray at some distance from the nozzle, therefore each data file typically contains measurements at a single axial coordinate and multiple transverse coordinates. However, there is no limitation in the file format on which position coordinates could be included in the file. This file format adds the capability to record multiple traces from multiple oscilloscopes (or other data acquisition devices), allowing us to record data in addition to the APD such as the injector driving current, dynamic pressure signal, etc.

Sample Raw Data File:

FileType=DataGrabberBinary X=0.40000 Y=-0.36000 NumberOfChannels=2 TimeStamp=2005-Oct-20_17:30:14 EPICS_1bmc:scaler1.S1=10000000 EPICS_1bmc:scaler1.S2=161869
Channel=0 UserDescription=APD DAQDevice=YokogawaScope_DL7480_Ch0 NumAverages=64 RecordLength=1000000 FirstPointTime=0.0 TimeStep=0.009615384615384616 DynamicRangeBits=8 BinaryDataType=short Volts=Scale*ADCValue/12+Offset Scale=0.0 Offset=0.0
*** 1000000 big-endian shorts here ***

Channel=1 UserDescription=GenotecCurrent DAQDevice=YokogawaScope_DL7480_Ch1 NumAverages=1 RecordLength=10000 FirstPointTime=0.0 TimeStep=1.0E-7 DynamicRangeBits=8 BinaryDataType=short Volts=Scale*ADCValue/12+Offset Scale=1.0 Offset=0.0
*** 10000 big-endian shorts here ***


FileType=DataGrabberBinary X=0.40000 Y=-0.33000 NumberOfChannels=2 TimeStamp=2005-Oct-20_17:30:23 EPICS_1bmc:scaler1.S1=10000000 EPICS_1bmc:scaler1.S2=161869
Channel=0 UserDescription=APD DAQDevice=YokogawaScope_DL7480_Ch0 NumAverages=64 RecordLength=1000000 FirstPointTime=0.0 TimeStep=0.009615384615384616 DynamicRangeBits=8 BinaryDataType=short Volts=Scale*ADCValue/12+Offset Scale=0.0 Offset=0.0
*** 1000000 big-endian shorts here ***

Channel=1 UserDescription=GenotecCurrent DAQDevice=YokogawaScope_DL7480_Ch1 NumAverages=1 RecordLength=10000 FirstPointTime=0.0 TimeStep=1.0E-7 DynamicRangeBits=8 BinaryDataType=short Volts=Scale*ADCValue/12+Offset Scale=1.0 Offset=0.0
*** 10000 big-endian shorts here ***

Explanation of File Format:

The file is broken up into any number of sections, each section is devoted to one particular X,Y position coordinate. At each position coordinate, any number of channels of data are recorded. The sections for each position coordinate are listed one after another, and are separated by a blank line.
Each position coordinate begins with an ASCII text header. This text header contains the x and y coordinate of the measurement, the number of data channels for this X,Y value, and any number of other parameters that are common to all data recorded at this coordinate. The header consists of key=value pairs separated by spaces, and is concluded with an end-of-line character. The key/value pairs often include scalar values that record parameters of the measurement, such as temperatures and pressures. There can be any number of key/value pairs and they can be in any order. The following key/value pairs are required in the coordinate header:

FileType=DataGrabberBinary
X=x
Y=y
NumberOfChannels=n

Following the coordinate header, there is a sub-section for each channel of data. The number of sub-sections is specified by the "NumberOfChannels" key/value from the coordinate header.

Each channel sub-section begins with an ASCII text header. This text header contains the channel number, the record length, the binary data type, and any number of other parameters. The header consists of key=value pairs separated by spaces, and is concluded with an end-of-line character. There can be any number of key/value pairs and they can be in any order. The following key/value pairs are required in the channel header:

Channel=n
RecordLength=m
BinaryDataType=type

The key/value UserDescription=description is used in the channel header to indicate the data stored in each channel. For example, UserDescription=APD is used to indicate that this particular channel holds the APD data. Our data analysis software looks for this particular key/value in order to recognize which channel contains the x-ray intensity as measured by the APD.
Following the channel header, the data for this channel is written as big-endian binary of the type specified in the header. For example, if the header for this channel specifies "RecordLength=10000 BinaryDataType=short", then there will be 10000 shorts. The data indicates the ADC value of the data acquisition device. To convert to volts, the conversion equation and factors may also be specified in the channel header.
Each data channel is separated by a blank line (end-of-line character).

Additional sections for additional coordinates follow in the order they were measured.

Experiments using Cornell PAD

Experiments that used this file format: Delphi1, Bosch5, Delphi2, Bosch9, Delphi5, Bosch17, Visteon2

Experiments done in Collaboration with Cornell have used their Pixel Array Detector. This detector uses software written by Cornell personnel to write data on a PC. The file format used to store the PAD data has evolved over time as the software has been improved. Descriptions of these file types are currently beyond the scope of this document.

Experiments using CCD camera for absorption imaging

Experiments that used this file format: Bosch4

In the Bosch4 experiment we used an Andor CCD camera to record absorption images of the spray. The images were copied from camera to PC using the Andor software and stored in that software's proprietary format. The description of these files is beyond the scope of this document.

Experiments using phase contrast imaging techniques

Experiments that used this file format: Bosch16

Phase-contrast imaging experiments use a CCD camera to store images, and PC-based software to transfer data from camera to disk. Kamel Fezzaa of APS knows the details of the camera, software, and file formats.