ayx_python_sdk.providers.amp_provider.data_transport package

Submodules

ayx_python_sdk.providers.amp_provider.data_transport.amp_transport module

Arrows stream formatted transport interface.

class ayx_python_sdk.providers.amp_provider.data_transport.amp_transport.AmpStreamTransport[source]

Bases: TransportBase

Transport interface to deliver data in arrows stream format.

get_rec_stream(record_batch: RecordBatch) Buffer[source]

Get an arrows formatted stream buffer.

iter_stream(stream: Buffer) Generator[Tuple[bytes, bool], None, None][source]

Break up a given stream by set chunk size.

logger = <RootLogger root (WARNING)>
receive_record(record_in_msg: RecordTransferIn) pa.Table[source]

Receive RecordTransferIn messages.

Returns None if the data received is only a chunk of a full record. Returns pa.Table if end_of_chunk is True.

send_record(record_batch: pa.RecordBatch, anchor_name: str, connection_name: str = '') Generator[RecordTransferOut, None, None][source]

Convert a given record batch into a sequence of RecordTransferOut msgs.

ayx_python_sdk.providers.amp_provider.data_transport.transport_base module

Transport implementation.

Transports provide the functionality for data management and IO. An implementation should provide an independant way to receive, parse, serialize, and send data to another service.

class ayx_python_sdk.providers.amp_provider.data_transport.transport_base.TransportBase[source]

Bases: ABC

Transport Manager for the tool.

abstract receive_record() None[source]

Pull data from source.

abstract send_record() None[source]

Push data to destination.

Module contents

Implementations of TransportBase.

class ayx_python_sdk.providers.amp_provider.data_transport.AmpStreamTransport[source]

Bases: TransportBase

Transport interface to deliver data in arrows stream format.

get_rec_stream(record_batch: RecordBatch) Buffer[source]

Get an arrows formatted stream buffer.

iter_stream(stream: Buffer) Generator[Tuple[bytes, bool], None, None][source]

Break up a given stream by set chunk size.

logger = <RootLogger root (WARNING)>
receive_record(record_in_msg: RecordTransferIn) pa.Table[source]

Receive RecordTransferIn messages.

Returns None if the data received is only a chunk of a full record. Returns pa.Table if end_of_chunk is True.

send_record(record_batch: pa.RecordBatch, anchor_name: str, connection_name: str = '') Generator[RecordTransferOut, None, None][source]

Convert a given record batch into a sequence of RecordTransferOut msgs.

stream_buffer: dict
class ayx_python_sdk.providers.amp_provider.data_transport.TransportBase[source]

Bases: ABC

Transport Manager for the tool.

abstract receive_record() None[source]

Pull data from source.

abstract send_record() None[source]

Push data to destination.