A data log for high-speed writing of data values. More...

#include <wpi/DataLog.h>

Inheritance diagram for wpi::log::DataLog:

Classes
class	Buffer

Public Member Functions
virtual	~DataLog ()=default

	DataLog (const DataLog &)=delete

DataLog &	operator= (const DataLog &)=delete

	DataLog (DataLog &&)=delete

DataLog &	operator= (const DataLog &&)=delete

virtual void	Flush ()=0
	Explicitly flushes the log data to disk. More...

virtual void	Pause ()
	Pauses appending of data records to the log. More...

virtual void	Resume ()
	Resumes appending of data records to the log. More...

virtual void	Stop ()
	Stops appending start/metadata/schema records to the log. More...

bool	HasSchema (std::string_view name) const
	Returns whether there is a data schema already registered with the given name. More...

void	AddSchema (std::string_view name, std::string_view type, std::span< const uint8_t > schema, int64_t timestamp=0)
	Registers a data schema. More...

void	AddSchema (std::string_view name, std::string_view type, std::string_view schema, int64_t timestamp=0)
	Registers a data schema. More...

template<ProtobufSerializable T>
void	AddProtobufSchema (ProtobufMessage< T > &msg, int64_t timestamp=0)
	Registers a protobuf schema. More...

template<typename T , typename... I> requires StructSerializable<T, I...>
void	AddStructSchema (const I &... info, int64_t timestamp=0)
	Registers a struct schema. More...

int	Start (std::string_view name, std::string_view type, std::string_view metadata={}, int64_t timestamp=0)
	Start an entry. More...

void	Finish (int entry, int64_t timestamp=0)
	Finish an entry. More...

void	SetMetadata (int entry, std::string_view metadata, int64_t timestamp=0)
	Updates the metadata for an entry. More...

void	AppendRaw (int entry, std::span< const uint8_t > data, int64_t timestamp)
	Appends a raw record to the log. More...

void	AppendRaw2 (int entry, std::span< const std::span< const uint8_t > > data, int64_t timestamp)
	Appends a raw record to the log. More...

void	AppendBoolean (int entry, bool value, int64_t timestamp)
	Appends a boolean record to the log. More...

void	AppendInteger (int entry, int64_t value, int64_t timestamp)
	Appends an integer record to the log. More...

void	AppendFloat (int entry, float value, int64_t timestamp)
	Appends a float record to the log. More...

void	AppendDouble (int entry, double value, int64_t timestamp)
	Appends a double record to the log. More...

void	AppendString (int entry, std::string_view value, int64_t timestamp)
	Appends a string record to the log. More...

void	AppendBooleanArray (int entry, std::span< const bool > arr, int64_t timestamp)
	Appends a boolean array record to the log. More...

void	AppendBooleanArray (int entry, std::span< const int > arr, int64_t timestamp)
	Appends a boolean array record to the log. More...

void	AppendBooleanArray (int entry, std::span< const uint8_t > arr, int64_t timestamp)
	Appends a boolean array record to the log. More...

void	AppendIntegerArray (int entry, std::span< const int64_t > arr, int64_t timestamp)
	Appends an integer array record to the log. More...

void	AppendFloatArray (int entry, std::span< const float > arr, int64_t timestamp)
	Appends a float array record to the log. More...

void	AppendDoubleArray (int entry, std::span< const double > arr, int64_t timestamp)
	Appends a double array record to the log. More...

void	AppendStringArray (int entry, std::span< const std::string > arr, int64_t timestamp)
	Appends a string array record to the log. More...

void	AppendStringArray (int entry, std::span< const std::string_view > arr, int64_t timestamp)
	Appends a string array record to the log. More...

void	AppendStringArray (int entry, std::span< const struct WPI_String > arr, int64_t timestamp)
	Appends a string array record to the log. More...

Protected Member Functions
	DataLog (wpi::Logger &msglog, std::string_view extraHeader="")
	Constructs the log. More...

void	StartFile ()
	Starts the log. More...

void	FlushBufs (std::vector< Buffer > *writeBufs)
	Provides complete set of all buffers that need to be written. More...

void	ReleaseBufs (std::vector< Buffer > *bufs)
	Releases memory for a set of buffers back to the internal buffer pool. More...

virtual void	BufferHalfFull ()
	Called when internal buffers are half the maximum count. More...

virtual bool	BufferFull ()=0
	Called when internal buffers reach the maximum count. More...

Protected Attributes
wpi::Logger &	m_msglog

Static Protected Attributes
static constexpr size_t	kBlockSize = 16 * 1024

static wpi::Logger	s_defaultMessageLog

Detailed Description

A data log for high-speed writing of data values.

The lifetime of the data log object must be longer than any data log entry objects that refer to it.

Finish() is needed only to indicate in the log that a particular entry is no longer being used (it releases the name to ID mapping). Finish() is not required to be called for data to be flushed to disk; entries in the log are written as Append() calls are being made. In fact, Finish() does not need to be called at all; this is helpful to avoid shutdown races where the DataLog object might be destroyed before other objects. It's often not a good idea to call Finish() from destructors for this reason.

DataLog calls are thread safe. DataLog uses a typical multiple-supplier, single-consumer setup. Writes to the log are atomic, but there is no guaranteed order in the log when multiple threads are writing to it; whichever thread grabs the write mutex first will get written first. For this reason (as well as the fact that timestamps can be set to arbitrary values), records in the log are not guaranteed to be sorted by timestamp.

Constructor & Destructor Documentation

◆ ~DataLog()

virtual wpi::log::DataLog::~DataLog ( )

virtualdefault

◆ DataLog() [1/3]

wpi::log::DataLog::DataLog ( const DataLog & )

delete

◆ DataLog() [2/3]

wpi::log::DataLog::DataLog ( DataLog && )

delete

◆ DataLog() [3/3]

wpi::log::DataLog::DataLog	(	wpi::Logger &	msglog,
		std::string_view	extraHeader = `""`
	)

inlineexplicitprotected

Constructs the log.

StartFile() must be called to actually start the file output.

Parameters

msglog	message logger (will be called from separate thread)
extraHeader	extra header metadata

Member Function Documentation

◆ AddProtobufSchema()

template<ProtobufSerializable T>

void wpi::log::DataLog::AddProtobufSchema	(	ProtobufMessage< T > &	msg,
		int64_t	timestamp = `0`
	)

inline

Registers a protobuf schema.

Duplicate calls to this function with the same name are silently ignored.

Template Parameters

T	protobuf serializable type

Parameters

msg	protobuf message
timestamp	Time stamp (0 to indicate now)

◆ AddSchema() [1/2]

void wpi::log::DataLog::AddSchema	(	std::string_view	name,
		std::string_view	type,
		std::span< const uint8_t >	schema,
		int64_t	timestamp = `0`
	)

Registers a data schema.

Data schemas provide information for how a certain data type string can be decoded. The type string of a data schema indicates the type of the schema itself (e.g. "protobuf" for protobuf schemas, "struct" for struct schemas, etc). In the data log, schemas are saved just like normal records, with the name being generated from the provided name: "/.schema/<name>". Duplicate calls to this function with the same name are silently ignored.

Parameters

name	Name (the string passed as the data type for records using this schema)
type	Type of schema (e.g. "protobuf", "struct", etc)
schema	Schema data
timestamp	Time stamp (may be 0 to indicate now)

◆ AddSchema() [2/2]

void wpi::log::DataLog::AddSchema	(	std::string_view	name,
		std::string_view	type,
		std::string_view	schema,
		int64_t	timestamp = `0`
	)

inline

Registers a data schema.

Data schemas provide information for how a certain data type string can be decoded. The type string of a data schema indicates the type of the schema itself (e.g. "protobuf" for protobuf schemas, "struct" for struct schemas, etc). In the data log, schemas are saved just like normal records, with the name being generated from the provided name: "/.schema/<name>". Duplicate calls to this function with the same name are silently ignored.

Parameters

name	Name (the string passed as the data type for records using this schema)
type	Type of schema (e.g. "protobuf", "struct", etc)
schema	Schema data
timestamp	Time stamp (may be 0 to indicate now)

◆ AddStructSchema()

template<typename T , typename... I>
requires StructSerializable<T, I...>

void wpi::log::DataLog::AddStructSchema	(	const I &...	info,
		int64_t	timestamp = `0`
	)

inline

Registers a struct schema.

Duplicate calls to this function with the same name are silently ignored.

Template Parameters

T	struct serializable type

Parameters

info	optional struct type info
timestamp	Time stamp (0 to indicate now)

◆ AppendBoolean()

void wpi::log::DataLog::AppendBoolean	(	int	entry,
		bool	value,
		int64_t	timestamp
	)

Appends a boolean record to the log.

Parameters

entry	Entry index, as returned by Start()
value	Boolean value to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendBooleanArray() [1/3]

void wpi::log::DataLog::AppendBooleanArray	(	int	entry,
		std::span< const bool >	arr,
		int64_t	timestamp
	)

Appends a boolean array record to the log.

Parameters

entry	Entry index, as returned by Start()
arr	Boolean array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendBooleanArray() [2/3]

void wpi::log::DataLog::AppendBooleanArray	(	int	entry,
		std::span< const int >	arr,
		int64_t	timestamp
	)

Appends a boolean array record to the log.

Parameters

entry	Entry index, as returned by Start()
arr	Boolean array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendBooleanArray() [3/3]

void wpi::log::DataLog::AppendBooleanArray	(	int	entry,
		std::span< const uint8_t >	arr,
		int64_t	timestamp
	)

Appends a boolean array record to the log.

Parameters

entry	Entry index, as returned by Start()
arr	Boolean array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendDouble()

void wpi::log::DataLog::AppendDouble	(	int	entry,
		double	value,
		int64_t	timestamp
	)

Appends a double record to the log.

Parameters

entry	Entry index, as returned by Start()
value	Double value to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendDoubleArray()

void wpi::log::DataLog::AppendDoubleArray	(	int	entry,
		std::span< const double >	arr,
		int64_t	timestamp
	)

Appends a double array record to the log.

Parameters

entry	Entry index, as returned by Start()
arr	Double array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendFloat()

void wpi::log::DataLog::AppendFloat	(	int	entry,
		float	value,
		int64_t	timestamp
	)

Appends a float record to the log.

Parameters

entry	Entry index, as returned by Start()
value	Float value to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendFloatArray()

void wpi::log::DataLog::AppendFloatArray	(	int	entry,
		std::span< const float >	arr,
		int64_t	timestamp
	)

Appends a float array record to the log.

Parameters

entry	Entry index, as returned by Start()
arr	Float array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendInteger()

void wpi::log::DataLog::AppendInteger	(	int	entry,
		int64_t	value,
		int64_t	timestamp
	)

Appends an integer record to the log.

Parameters

entry	Entry index, as returned by Start()
value	Integer value to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendIntegerArray()

void wpi::log::DataLog::AppendIntegerArray	(	int	entry,
		std::span< const int64_t >	arr,
		int64_t	timestamp
	)

Appends an integer array record to the log.

Parameters

entry	Entry index, as returned by Start()
arr	Integer array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendRaw()

void wpi::log::DataLog::AppendRaw	(	int	entry,
		std::span< const uint8_t >	data,
		int64_t	timestamp
	)

Appends a raw record to the log.

Parameters

entry	Entry index, as returned by Start()
data	Byte array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendRaw2()

void wpi::log::DataLog::AppendRaw2	(	int	entry,
		std::span< const std::span< const uint8_t > >	data,
		int64_t	timestamp
	)

Appends a raw record to the log.

Parameters

entry	Entry index, as returned by Start()
data	Byte array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendString()

void wpi::log::DataLog::AppendString	(	int	entry,
		std::string_view	value,
		int64_t	timestamp
	)

Appends a string record to the log.

Parameters

entry	Entry index, as returned by Start()
value	String value to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendStringArray() [1/3]

void wpi::log::DataLog::AppendStringArray	(	int	entry,
		std::span< const std::string >	arr,
		int64_t	timestamp
	)

Appends a string array record to the log.

Parameters

entry	Entry index, as returned by Start()
arr	String array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendStringArray() [2/3]

void wpi::log::DataLog::AppendStringArray	(	int	entry,
		std::span< const std::string_view >	arr,
		int64_t	timestamp
	)

Appends a string array record to the log.

Parameters

entry	Entry index, as returned by Start()
arr	String array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ AppendStringArray() [3/3]

void wpi::log::DataLog::AppendStringArray	(	int	entry,
		std::span< const struct WPI_String >	arr,
		int64_t	timestamp
	)

Appends a string array record to the log.

Parameters

entry	Entry index, as returned by Start()
arr	String array to record
timestamp	Time stamp (may be 0 to indicate now)

◆ BufferFull()

virtual bool wpi::log::DataLog::BufferFull ( )

protectedpure virtual

Called when internal buffers reach the maximum count.

Called with internal mutex held; do not call any other DataLog functions from this function.

Returns: true if log should be paused (don't call PauseLog)

◆ BufferHalfFull()

virtual void wpi::log::DataLog::BufferHalfFull ( )

protectedvirtual

Called when internal buffers are half the maximum count.

Called with internal mutex held; do not call any other DataLog functions from this function.

◆ Finish()

void wpi::log::DataLog::Finish	(	int	entry,
		int64_t	timestamp = `0`
	)

Finish an entry.

Parameters

entry	Entry index
timestamp	Time stamp (may be 0 to indicate now)

◆ Flush()

virtual void wpi::log::DataLog::Flush ( )

pure virtual

Explicitly flushes the log data to disk.

Implemented in wpi::log::DataLogBackgroundWriter, and wpi::log::DataLogWriter.

◆ FlushBufs()

void wpi::log::DataLog::FlushBufs ( std::vector< Buffer > * writeBufs )

protected

Provides complete set of all buffers that need to be written.

Any existing contents of writeBufs will be released as if ReleaseBufs() was called prior to this call.

Returned buffers should provided back via ReleaseBufs() after the write is complete.

Parameters

writeBufs buffers to be written (output)

◆ HasSchema()

bool wpi::log::DataLog::HasSchema ( std::string_view name ) const

Returns whether there is a data schema already registered with the given name.

Parameters

name	Name (the string passed as the data type for records using this schema)

Returns: True if schema already registered

◆ operator=() [1/2]

DataLog & wpi::log::DataLog::operator= ( const DataLog && )

delete

◆ operator=() [2/2]

DataLog & wpi::log::DataLog::operator= ( const DataLog & )

delete

◆ Pause()

virtual void wpi::log::DataLog::Pause ( )

virtual

Pauses appending of data records to the log.

While paused, no data records are saved (e.g. AppendX is a no-op). Has no effect on entry starts / finishes / metadata changes.

Reimplemented in wpi::log::DataLogBackgroundWriter.

◆ ReleaseBufs()

void wpi::log::DataLog::ReleaseBufs ( std::vector< Buffer > * bufs )

protected

Releases memory for a set of buffers back to the internal buffer pool.

Parameters

bufs	buffers; empty on return

◆ Resume()

virtual void wpi::log::DataLog::Resume ( )

virtual

Resumes appending of data records to the log.

Reimplemented in wpi::log::DataLogBackgroundWriter.

◆ SetMetadata()

void wpi::log::DataLog::SetMetadata	(	int	entry,
		std::string_view	metadata,
		int64_t	timestamp = `0`
	)

Updates the metadata for an entry.

Parameters

entry	Entry index
metadata	New metadata for the entry
timestamp	Time stamp (may be 0 to indicate now)

◆ Start()

int wpi::log::DataLog::Start	(	std::string_view	name,
		std::string_view	type,
		std::string_view	metadata = `{}`,
		int64_t	timestamp = `0`
	)

Start an entry.

Duplicate names are allowed (with the same type), and result in the same index being returned (Start/Finish are reference counted). A duplicate name with a different type will result in an error message being printed to the console and 0 being returned (which will be ignored by the Append functions).

Parameters

name	Name
type	Data type
metadata	Initial metadata (e.g. data properties)
timestamp	Time stamp (may be 0 to indicate now)

Returns: Entry index

◆ StartFile()

void wpi::log::DataLog::StartFile ( )

protected

Starts the log.

Appends file header and Start records and schema data values for all previously started entries and schemas. No effect unless the data log is currently stopped.

◆ Stop()

virtual void wpi::log::DataLog::Stop ( )

virtual

Stops appending start/metadata/schema records to the log.

Reimplemented in wpi::log::DataLogBackgroundWriter, and wpi::log::DataLogWriter.

Member Data Documentation

◆ kBlockSize

constexpr size_t wpi::log::DataLog::kBlockSize = 16 * 1024

staticconstexprprotected

◆ m_msglog

wpi::Logger& wpi::log::DataLog::m_msglog

protected

◆ s_defaultMessageLog

wpi::Logger wpi::log::DataLog::s_defaultMessageLog

staticprotected

The documentation for this class was generated from the following file:

wpi/DataLog.h

Classes

Public Member Functions

Protected Member Functions

Protected Attributes

Static Protected Attributes

Detailed Description

Constructor & Destructor Documentation

◆ ~DataLog()

◆ DataLog() [1/3]

◆ DataLog() [2/3]

◆ DataLog() [3/3]

Member Function Documentation

◆ AddProtobufSchema()

◆ AddSchema() [1/2]

◆ AddSchema() [2/2]

◆ AddStructSchema()

◆ AppendBoolean()

◆ AppendBooleanArray() [1/3]

◆ AppendBooleanArray() [2/3]

◆ AppendBooleanArray() [3/3]

◆ AppendDouble()

◆ AppendDoubleArray()

◆ AppendFloat()

◆ AppendFloatArray()

◆ AppendInteger()

◆ AppendIntegerArray()

◆ AppendRaw()

◆ AppendRaw2()

◆ AppendString()

◆ AppendStringArray() [1/3]

◆ AppendStringArray() [2/3]

◆ AppendStringArray() [3/3]

◆ BufferFull()

◆ BufferHalfFull()

◆ Finish()

◆ Flush()

◆ FlushBufs()

◆ HasSchema()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ Pause()

◆ ReleaseBufs()

◆ Resume()

◆ SetMetadata()

◆ Start()

◆ StartFile()

◆ Stop()

Member Data Documentation

◆ kBlockSize

◆ m_msglog

◆ s_defaultMessageLog