Module src.make_pipe

Functions

def make_api_pipe(api_name: str, writer: collections.abc.Callable, sleeping_time: float = None)

pipe maker for Api requests

Arguments

api_name(Required):

String.
Api identifier bound to a Config name in ConfigPath (check Pipeline doc)

ConfigPath(
    Config(name="random_name".......)
)

"random_name"  <-- api_name

writer(Required):

Callable which take an List(request.requests)  in argument.
Function which override the ApiPipeline.write method (check ApiPipeline doc)

sleeping_time(Optional):

Float.
The time in second to sleep between 2 requests (check Pipeline doc)

Return

pipe_instance:

An Instance of pipe (an ApiPipeline subclass with 'writer' arg that override ApiPipeline.write method)

def run_api_pipe(pipe_instance, request_arguments: List[tuple], retry_fails: bool = False, transaction_rate: int = None)

pip runner to execute a pipe instance

Arguments

pipe_instance(Required):

An Instance of pipe (an ApiPipeline subclass with 'writer' arg that override ApiPipeline.write method)
call make_api_pipe function to get a pipe_instance (see the doc above)

request_arguments(Required):

a list of 2-tuple elements like ("end_url", {"param_name": "param_val"})
to provide request configurations check the RequestFactory doc

retry_fails(Optional):

Boolean.
if put to 'True', request which failed will be executed a second time

transaction_rate(Optional):

Int.
The number of request results to be passed to the pipe.write method each time
if not configured, write method is executed once after all pipe requests processed
check Pipeline Documentation

Return

err_log:

a log where are stored all requests informations which failed
(2 times if 'retry_fails' argument is set to True)