Astrophysics (Index)About

pipeline

(set series of tasks to perform on something)

The term pipeline is used in technology, including research astronomy for a sequence of processing steps carried out on data, especially if data is passed through from step to step such that when data is passed from step 1 to step 2, additional data is taken up for step 1 so the two can run simultaneously, and analogously for all the steps. Thus a stream of data is being processed, at any instant, some by each step. The term may be used to indicate and explain this simultaneous-yet-sequential processing, but is also used sometimes without the literal intent, but merely to explain the order of the processing of the data.

Literal pipelines (as more than an explaining notion) are a candidate when there are certain kinds of challenges in processing the data: to finish the processing for some data quickly and/or to tackle a huge amount of data and/or to organize some complicated processing. A prime circumstance is a case such that observation provides too much data to save and considerable processing is needed to reduce it to the point that it can be saved. This occurs in surveys of transient phenomena, often done with automated telescopes. They create something like a high-resolution digitized time-lapse movie of the sky, possibly even a data cube per frame, but the information of interest is undiscovered astronomical transients such as supernovae or near-Earth objects, rather than things like Earth weather or passing airplanes. The challenge in this case a way to classify the observed change promptly and open the possibility for prompt follow-up observation, and last part of the pipeline might consist of sending alerts to appropriate astronomers. Pipelines are ubiquitous for such surveys and the design, deployment, and operation of an effective pipeline can require a significant part of the resources dedicated to the survey. The concept of a pipeline is also a good description of typical radio astronomy observation/processing including radio interferometry.

Pipelines can also be an efficient way to handle huge sets of data to be processed even if there is not such an urgent deadline. Some computing can be looked upon as a pipeline or not, according to the manner in which the researcher decides to describe the process.

The pipeline steps may be computer programs running on a single computer, or on interconnected computers. It may begin with analog electronics. Data communication can be a part of it, e.g., specifying how data will be transferred from computers near the telescope to those at some distant location more suitable for computer processing. Steps might also involve the actions of people, they may be involved along the way, such as to verify or filter, and carry out other steps by hand though often the goal is to automate, and the "pipeline-like description" may describe to some degree actions of astronomers notified by an alert.


The term pipeline is also used in science and technology for other circumstances where something or someone is passed from step to step, such as the education of astronomers/astrophysicists or the development of projects.


(computation,automated)
Further reading:
https://arxiv.org/abs/1810.06012
https://arxiv.org/abs/1510.04331

Index