Log Wire Codec

Data Structures
struct	LogWireRecord
struct	LogSubSpec
struct	LogWireChanInfo
	One channel in a sender's catalog. More...
struct	LogWireCatalog
struct	LogWireGap
struct	LogWireFrame

Typedefs
typedef struct LogWireRecord	LogWireRecord
typedef struct LogSubSpec	LogSubSpec
typedef struct LogWireChanInfo	LogWireChanInfo
	One channel in a sender's catalog.
typedef struct LogWireCatalog	LogWireCatalog
typedef struct LogWireGap	LogWireGap
typedef struct LogWireFrame	LogWireFrame
typedef bool(*	LogWireFrameCB) (const LogWireFrame *frame, void *ctx)
typedef struct LogWireEncoder	LogWireEncoder
	Opaque encoder; one per connection.
typedef struct LogWireDecoder	LogWireDecoder
	Opaque decoder; one per connection.

Enumerations
enum	LOG_WIRE_FRAME {   LOG_WireSegment = 1 , LOG_WireChanDecl , LOG_WireSiteDecl , LOG_WireEntry ,   LOG_WireGap , LOG_WireSubscribe , LOG_WireCatalog }
enum	LOG_WIRE_FLAGS { LOG_WireOmitCtx = 0x00000001 }
	Encoder options. More...

Functions
LogWireEncoder *	logWireEncoderCreate (strref origin, flags_t flags)
void	logWireEncoderDestroy (LogWireEncoder **enc)
bool	logWireEncode (LogWireEncoder *enc, Buffer *out, const LogRecord *rec)
bool	logWireEncodeGap (LogWireEncoder *enc, Buffer *out, uint64 n, uint64 firstseq, uint64 lastseq)
bool	logWireEncodeSub (LogWireEncoder *enc, Buffer *out, const LogSubSpec *spec)
bool	logWireEncodeCatalog (LogWireEncoder *enc, Buffer *out, _In_reads_(nchans) const LogWireChanInfo *chans, int nchans)
void	logWireEndSegment (LogWireEncoder *enc)
LogWireDecoder *	logWireDecoderCreate (void)
void	logWireDecoderDestroy (LogWireDecoder **dec)
bool	logWireDecode (LogWireDecoder *dec, const uint8 *buf, size_t len, LogWireFrameCB cb, void *ctx)
bool	logInject (strref chanpath, const LogWireRecord *rec)

Detailed Description

Turns a LogRecord into bytes that another cx instance can turn back into a LogRecord.

cx never touches a wire. There is no listening, connecting, reading, writing or authenticating anywhere in the log system. This codec produces complete frames and hands them to whoever asked for them; the application decides what a connection is, and may multiplex log traffic into a stream it already has, wrap it in its own authentication, or carry it over something that is not a socket at all. See Forwarding for the destination that drives this codec, and logInject() for the receiving end.

Frames and segments

A frame is a kind, a length, and a payload. Frames arrive in order and a decoder hands each one to a callback as it completes, so a transport may deliver bytes in any sized pieces.

Frames are grouped into segments. A segment is one self-contained document: everything a frame refers to – channel paths, message templates, field names – is declared once within the segment that uses it, so a busy channel costs its path exactly once no matter how many records name it. A new segment starts whenever the encoder is told to end the current one, which is what makes a stored run of frames safe to trim from the front: whole segments can be dropped and the rest still decodes.

LogWireEncoder *enc = logWireEncoderCreate(_SL("web-01"), 0);
 
Buffer frames = 0;
logWireEncode(enc, &frames, rec);   // frames now holds one or more complete frames
...send them...
 
bufDestroy(&frames);
logWireEncoderDestroy(&enc);

Output buffers

Every encode call writes its frames into a Buffer, replacing whatever was in it. Hand the same Buffer back on the next call and it is reused, growing only when a run of frames needs more room than it already has. Destroy it with bufDestroy() when you are done with it.

Typedef Documentation

LogSubSpec

typedef struct LogSubSpec LogSubSpec

What a receiver has asked a sender for

A sender ships nothing until one of these arrives; there is no locally configured "forward everything to host X". See Forwarding.

LogWireCatalog

typedef struct LogWireCatalog LogWireCatalog

What a sender is capable of logging

The channel inventory, which is what an operator browses before deciding what to subscribe to. Call sites are not in it: C offers no portable way to enumerate the statics in a binary, so what a channel actually says only becomes visible once something logs to it.

LogWireFrame

typedef struct LogWireFrame LogWireFrame

One decoded frame

Only the member matching kind is filled in; the rest are NULL.

LogWireFrameCB

typedef bool(* LogWireFrameCB) (const LogWireFrame *frame, void *ctx)

Called once per complete frame

Parameters

frame	The frame; valid only for the duration of the call
ctx	User context passed to logWireDecode()

Returns

false to stop decoding, which fails the whole logWireDecode() call

Definition at line 162 of file logwire.h.

LogWireGap

typedef struct LogWireGap LogWireGap

Records a sender dropped instead of sending

A gap says how much is missing and where, so a receiver can say so rather than silently showing a shorter log than the sender produced.

LogWireRecord

typedef struct LogWireRecord LogWireRecord

One decoded log record, as it came off the wire

Everything here is borrowed for the duration of the frame callback. Hand it to logInject() to deliver it locally, or copy what you need.

Enumeration Type Documentation

LOG_WIRE_FLAGS

enum LOG_WIRE_FLAGS

Encoder options.

Enumerator
LOG_WireOmitCtx	Leave context fields out of every record For a receiver that does its own correlation and does not want the sender's. The record's own arguments are unaffected.

Definition at line 69 of file logwire.h.

LOG_WIRE_FRAME

enum LOG_WIRE_FRAME

Kinds of frame that can appear on the wire

A decoder hands a frame kind it does not implement to the callback with nothing filled in, rather than failing, so an older receiver keeps working against a newer sender.

Enumerator
LOG_WireSegment	Opens a segment; everything declared in the old one is forgotten.
LOG_WireChanDecl	Declares a channel path and its policy, for this segment.
LOG_WireSiteDecl	Declares a call site's message template, for this segment.
LOG_WireEntry	One log record.
LOG_WireGap	Records that were dropped rather than sent.
LOG_WireSubscribe	A receiver asking for a set of channels and levels.
LOG_WireCatalog	What a sender is capable of logging.

Definition at line 58 of file logwire.h.

Function Documentation

logInject()

bool logInject	(	strref	chanpath,
		const LogWireRecord *	rec
	)

Deliver a record that came from another instance to this one's destinations

The record goes through the same routing, filtering and per-channel level checks as anything logged here, so a record that does not arrive was dropped for a reason a local record would have been dropped for too. Its timestamp, sequence number and batch id are the sender's and are not replaced.

Parameters

chanpath	Channel to deliver it to, interned locally if it is new
rec	Record decoded from the wire

Returns

false if the record was not delivered

logInject(frame->rec->chanpath, frame->rec);

logWireDecode()

bool logWireDecode	(	LogWireDecoder *	dec,
		const uint8 *	buf,
		size_t	len,
		LogWireFrameCB	cb,
		void *	ctx
	)

Feed received bytes to a decoder

Every frame that completes is handed to cb. A frame that is only partly here is kept until the rest of it arrives, so bytes may be fed in any sized pieces.

Parameters

dec	Decoder to feed
buf	Bytes received
len	Number of bytes
cb	Called once per complete frame
ctx	Passed to the callback

Returns

false if the stream is malformed. The decoder cannot be used again after that and must be destroyed; close the connection it came from.

if (!logWireDecode(dec, buf, len, myFrameCB, self))
    closeConnection(self);

logWireDecoderCreate()

LogWireDecoder * logWireDecoderCreate

(

void

)

Create a decoder

Returns

A new decoder; destroy with logWireDecoderDestroy()

logWireDecoderDestroy()

void logWireDecoderDestroy

(

LogWireDecoder **

dec

)

Destroy a decoder

Parameters

dec	Pointer to the decoder handle; set to NULL

logWireEncode()

bool logWireEncode	(	LogWireEncoder *	enc,
		Buffer *	out,
		const LogRecord *	rec
	)

Encode one record

Writes one or more complete frames into out: whatever declarations this record needs and has not sent yet, then the record itself. Nothing partial is ever written.

Parameters

enc	Encoder to use
out	Receives the frames, replacing anything already in the buffer. A NULL buffer is created; an existing one is reused and grown as needed.
rec	Record to encode

Returns

false if the record could not be encoded; out is left empty in that case

Buffer frames = 0;
logWireEncode(enc, &frames, rec);

logWireEncodeCatalog()

bool logWireEncodeCatalog	(	LogWireEncoder *	enc,
		Buffer *	out,
		_In_reads_(nchans) const LogWireChanInfo *	chans,
		int	nchans
	)

Encode a channel catalog

Parameters

enc	Encoder to use
out	Receives the frame, replacing anything already in the buffer
chans	Channels to describe
nchans	Number of channels

Returns

false if the frame could not be encoded

logWireEncodeGap()

bool logWireEncodeGap	(	LogWireEncoder *	enc,
		Buffer *	out,
		uint64	n,
		uint64	firstseq,
		uint64	lastseq
	)

Encode a gap

Parameters

enc	Encoder to use
out	Receives the frame, replacing anything already in the buffer
n	How many records were dropped
firstseq	Sequence number of the first one
lastseq	Sequence number of the last one

Returns

false if the frame could not be encoded

logWireEncoderCreate()

LogWireEncoder * logWireEncoderCreate	(	strref	origin,
		flags_t	flags
	)

Create an encoder

Parameters

origin	This instance's identity, stamped on every record that does not already carry one. Empty leaves records unstamped.
flags	LOG_WIRE_FLAGS options, or 0

Returns

A new encoder; destroy with logWireEncoderDestroy()

LogWireEncoder *enc = logWireEncoderCreate(_SL("web-01"), 0);

logWireEncoderDestroy()

void logWireEncoderDestroy

(

LogWireEncoder **

enc

)

Destroy an encoder

Parameters

enc	Pointer to the encoder handle; set to NULL

logWireEncodeSub()

bool logWireEncodeSub	(	LogWireEncoder *	enc,
		Buffer *	out,
		const LogSubSpec *	spec
	)

Encode a subscription request

For the receiving side of a connection: this is what a collector sends to a sender to say what it wants. Sending it needs an encoder of its own, since the control plane is the same framing running the other way.

Parameters

enc	Encoder to use
out	Receives the frame, replacing anything already in the buffer
spec	What is being asked for; NULL cancels an existing subscription

Returns

false if the frame could not be encoded

LogSubSpec spec = { .maxlevel = LOG_Info };
saInit(&spec.patterns, string, 1);
saPush(&spec.patterns, string, _S"app/**");
logWireEncodeSub(enc, &out, &spec);
saDestroy(&spec.patterns);

logWireEndSegment()

void logWireEndSegment

(

LogWireEncoder *

enc

)

End the current segment

The next record encoded opens a new one and re-sends the declarations it needs. Call this wherever a receiver may only ever see part of what was produced – before storing frames that might later be trimmed, and on reconnect.

Parameters

enc	Encoder to end the segment on