Converting PCAPs to Other Formats¶
The Ouster Python SDK provides several examples to convert PCAP files to other common point cloud formats such as CSV, LAS, PCD, and PLY.
PCAPs to CSV¶
Sometimes we want to get a point cloud (XYZ + other fields) as a CSV file for further
analysis with other tools.
To convert the first 5 frames of our sample data from a pcap file, you can try:
$ ouster-cli source --meta $SAMPLE_DATA_JSON_PATH $SAMPLE_DATA_PCAP_PATH slice 0:5 save output.csv
PS > ouster-cli.exe source --meta $SAMPLE_DATA_JSON_PATH $SAMPLE_DATA_PCAP_PATH slice 0:5 save output.csv
The following function implements the pcap to csv conversion above.
def source_to_csv_iter(frame_set_iter: Iterator[FrameSet], infos: List[SensorInfo],
prefix: str = "", dir: str = "", overwrite: bool = True,
filename: str = "") -> Iterable[FrameSet]:
"""Create a CSV saving iterator from a LidarFrame iterator
The number of saved lines per csv file is always H x W, which corresponds to
a full 2D image representation of a lidar frame.
Pixel fields with shape (H, W) are exported as one column each. Multi-channel
pixel fields (e.g. with shape (H, W, 3)) are exported with one column per
channel: field_R, field_G, field_B.
Each line in a csv file is (for DUAL profile):
TIMESTAMP (ns), ROW, DESTAGGERED IMAGE COLUMN, MEASUREMENT_ID,
B_ATTENUATED, B_CLEAR, G_ATTENUATED, G_CLEAR, NEAR_IR (photons),
RANGE (mm), RANGE2 (mm), SIGNAL (photons),
SIGNAL2 (photons), REFLECTIVITY (%), REFLECTIVITY2 (%),
R_ATTENUATED, R_CLEAR, NEAR_IR (photons),
field_R, field_G, field_B, X (m), Y (m), Z (m), X2 (m), Y2 (m), Z2(m)
"""
for info in infos:
if info.num_returns > 1:
print("Note: You've selected to convert a dual returns pcap to CSV. Each row "
"will represent a single pixel, so that both returns for that pixel will "
"be on a single row. As this is an example we provide for getting "
"started, we realize that you may have conversion needs which are not met "
"by this function. You can find the source code on the Python SDK "
"documentation website to modify it for your own needs.")
break
# Build filenames
filenames = []
for info in infos:
name = determine_filename(filename=filename, info=info, extension=".csv", prefix=prefix, dir=dir)
name = name[0:-4] # remove extension
filenames.append(name)
click.echo(f"Saving CSV files at {name}_XXX.csv")
create_directories_if_missing(filenames[0])
# Construct csv header and data format
def get_fields_info(frame: LidarFrame) -> Tuple[str, List[str]]:
field_names = 'TIMESTAMP (ns), ROW, DESTAGGERED IMAGE COLUMN, MEASUREMENT_ID'
field_fmts = ['%d'] * 4
dual = ChanField.RANGE2 in frame.fields
for chan_field in frame.fields:
if frame.field_class(chan_field) != FieldClass.PIXEL_FIELD:
continue
arr = frame.field(chan_field)
ndim = arr.ndim
pixel_fmt = '%.4f' if np.issubdtype(arr.dtype, np.floating) else '%d'
if ndim == 2:
field_names += f', {chan_field}'
if chan_field in [ChanField.RANGE, ChanField.RANGE2]:
field_names += ' (mm)'
if chan_field in [ChanField.REFLECTIVITY, ChanField.REFLECTIVITY2]:
field_names += ' (%)'
if chan_field in [ChanField.SIGNAL, ChanField.SIGNAL2,
ChanField.NEAR_IR]:
field_names += ' (photons)'
field_fmts.append(pixel_fmt)
elif ndim == 3:
n_channels = arr.shape[2]
if n_channels == 1:
suffixes: Tuple[str, ...] = ('',)
elif n_channels == 3:
if chan_field in [ChanField.NORMALS, ChanField.NORMALS2]:
suffixes = ('_X', '_Y', '_Z')
else:
suffixes = ('_R', '_G', '_B')
else:
suffixes = tuple(f'_{i}' for i in range(n_channels))
for suf in suffixes:
field_names += f', {chan_field}{suf}'
field_fmts.append(pixel_fmt)
else:
continue
field_names += ', X1 (m), Y1 (m), Z1 (m)'
field_fmts.extend(3 * ['%.4f'])
if dual:
field_names += ', X2 (m), Y2 (m), Z2 (m)'
field_fmts.extend(3 * ['%.4f'])
return field_names, field_fmts
field_names: Dict[int, str] = {}
field_fmts: Dict[int, List[str]] = {}
# {recompute xyzlut to save computation in a loop
xyzlut = []
row_layer = []
column_layer_staggered = []
for info in infos:
xyzlut.append(XYZLut(info, False))
row_layer.append(np.fromfunction(lambda i, j: i,
(info.format.pixels_per_column,
info.format.columns_per_frame), dtype=int))
column_layer = np.fromfunction(lambda i, j: j,
(info.format.pixels_per_column,
info.format.columns_per_frame), dtype=int)
column_layer_staggered.append(destagger(info, column_layer,
inverse=True))
saved = False
def save_iter():
nonlocal saved
try:
if saved:
for frame in frame_set_iter():
yield frame
return
for idx, frames in enumerate(frame_set_iter()):
for lidar_idx, frame in enumerate(frames):
if frame is None:
continue
# Initialize the field names for csv header
if lidar_idx not in field_names or lidar_idx not in field_fmts:
field_names[lidar_idx], field_fmts[lidar_idx] = get_fields_info(frame)
# Copy per-column timestamps and measurement_ids for each beam
timestamps = np.tile(frame.timestamp, (frame.h, 1))
measurement_ids = np.tile(frame.measurement_id, (frame.h, 1))
# Keep only per-pixel fields for CSV stacking.
csv_fields = [field for field in frame.fields
if frame.field_class(field) == FieldClass.PIXEL_FIELD]
# Grab channel data
fields_values = []
for ch in csv_fields:
arr = frame.field(ch)
if arr.ndim == 2:
fields_values.append(arr)
elif arr.ndim == 3:
for c in range(arr.shape[2]):
fields_values.append(arr[:, :, c])
else:
continue
lidar_frame = frame
frame = np.dstack((timestamps, row_layer[lidar_idx], column_layer_staggered[lidar_idx],
measurement_ids, *fields_values))
# Output points in "image" vs. staggered order
frame = destagger(info, frame)
# Destagger XYZ separately since it has a different type
xyz = xyzlut[lidar_idx](lidar_frame.field(ChanField.RANGE))
xyz_destaggered = destagger(info, xyz)
if ChanField.RANGE2 in csv_fields:
xyz2 = xyzlut[lidar_idx](lidar_frame.field(ChanField.RANGE2))
xyz2_destaggered = destagger(info, xyz2)
# Get all data as one H x W x num fields int64 array for savetxt()
frame = np.dstack(tuple(map(lambda x: x.astype(object),
(frame, xyz_destaggered, xyz2_destaggered))))
else:
# Get all data as one H x W x num fields int64 array for savetxt()
frame = np.dstack(tuple(map(lambda x: x.astype(object),
(frame, xyz_destaggered))))
frame_colmajor = np.swapaxes(frame, 0, 1)
# Write csv out to file
csv_path = f"{filenames[lidar_idx]}_s{lidar_idx}_{idx}.csv"
print(f'write frame index #{idx} sensor_idx #{lidar_idx}, to file: {csv_path}')
if os.path.isfile(csv_path) and not overwrite:
print(_file_exists_error(csv_path))
exit(1)
header = '\n'.join([f'frame num: {idx}', field_names[lidar_idx]])
np.savetxt(csv_path,
frame_colmajor.reshape(-1, frame.shape[2]),
fmt=field_fmts[lidar_idx],
delimiter=',',
header=header)
yield frame
except (KeyboardInterrupt, StopIteration):
pass
finally:
saved = True
# type ignored because generators are tricky to mypy
return save_iter # type: ignore
Because we stored the frame as structured 2D images, we can easily recover it by loading it back into
a numpy.ndarray and continuing to use it as a 2D image.
import numpy as np
# read array from CSV
frame = np.loadtxt('output_s0_0.csv', delimiter=',')
# convert back to "fat" 2D image [H x W x num_fields] shape
frame = frame.reshape((128, -1, frame.shape[1]))
We used 128 while restoring 2D image from a CSV file because it’s the number of channels of our
OS-1-128.pcap sample data recording.
PCAPs to LAS¶
To convert to the first 5 frames of our sample data from a pcap file to LAS, you can try:
$ python3 -m ouster.sdk.examples.pcap $SAMPLE_DATA_PCAP_PATH pcap-to-las --frame-num 5
PS > py -3 -m ouster.sdk.examples.pcap $SAMPLE_DATA_PCAP_PATH pcap-to-las --frame-num 5
Checkout the examples.pcap.pcap_to_las() documentation for the example source code.
PCAPs to PCD¶
To convert to the first 5 frames of our sample data from a pcap file to PCD, you can try:
$ python3 -m ouster.sdk.examples.pcap $SAMPLE_DATA_PCAP_PATH pcap-to-pcd --frame-num 5
PS > py -3 -m ouster.sdk.examples.pcap $SAMPLE_DATA_PCAP_PATH pcap-to-pcd --frame-num 5
Checkout the examples.pcap.pcap_to_pcd() documentation for the example source code.
PCAPs to PLY¶
Here we will reuse the PCAP to PCD function that uses Open3d and will exploit the extensive Open3d
File IO that gives us an easy way to save the loaded point cloud to PLY. Alternative ways are available via plyfile library.
To convert to the first 5 frames of our sample data from a pcap file to PLY, you can try:
$ python3 -m ouster.sdk.examples.pcap $SAMPLE_DATA_PCAP_PATH pcap-to-ply --frame-num 5
PS > py -3 -m ouster.sdk.examples.pcap $SAMPLE_DATA_PCAP_PATH pcap-to-ply --frame-num 5
Checkout the examples.pcap.pcap_to_ply() documentation for the example source code.