Telestream General Pipeline System Manual

Pipeline system architecture
Pipeline is a unique network attached encoder system that can be used in a
variety of capture and playout workflows. Being network attached there are
several architectural constraints that need to be understood in order to
configure an efficient and reliable Pipeline system.
Pipeline system capture workflows consists of taking baseband HD/SD-SDI
inputs, compressing it, creating a file with the compressed data and delivering
the completed file to a folder location. The Pipeline system will consists of one
or more Pipeline network attached encoder devices connected via CAT6
Ethernet to a Pipeline host computer.
Pipeline network attached encoder devices should be connected directly via
CAT6 Ethernet to a Pipeline host computer or via a fully non-blocking
managed Layer 2/3 Enterprise Edge Ethernet switch with adequate per-port
data buffering.
Pipeline System
Because of Pipeline’s unique network attached architecture it is important to
understand each component of a Pipeline system in order to create a system
that will meet the needs of your workflow.
A Pipeline System consists of one or more Pipeline channels and a Pipeline
host computer (Mac or Windows PC) connected together via Ethernet wiring.
1
General Pipeline System
Setup Information
Pipeline
Product Sheet

The key components of a Pipeline System are the
Pipeline device hardware, Pipeline network, the Pipeline
host hardware, the Pipeline application software and
the Pipeline media disk storage systems. Each of these
components is described below.
Pipeline device hardware
Pipeline device hardware has up to four independent
channels with SDI I/O for capture and play out. The SDI
connectors take baseband SD/HD-SDI signals as input,
compress them into one of many supported formats
and stream the compress data (video, audio and control
data) out their respective Ethernet ports using Real
Time Streaming Protocol (RTSP). Each Pipeline
channel’s RTSP stream contains video, audio and appli-
cation/control data in RTP transport packets. These
Pipeline channel streams are consumed by Pipeline
application software running on a Pipeline host
computer. The Pipeline application software will wrap
each RTSP stream into the appropriate file wrapper and
deliver these files to their respective media storage
locations.
Pipeline host and Pipeline application software
The Pipeline host computer is primarily responsible for
taking one or more Pipeline channel’s RTSP streams
via Ethernet, wrapping them in the appropriate format
and writing the resulting file to the selected media
destination. Pipeline host systems can be Mac OS X or
Microsoft Windows based.
Note: Pipeline Control software captures data
coming from one or more Pipeline channels, and
writes this data to disk in real-time. To prevent
interference with Pipeline Control any process, such
as real-time anti-virus scanning, should be disabled
while Pipeline Control is in use. Other processes
that periodically run such as operating system
updates should also be disabled.
A Pipeline host computer can control multiple Pipeline
channels concurrently. Use this document to under-
stand how to configure a Pipeline System to meet your
particular needs.
Pipeline software applications perform many tasks.
These task range from scheduled capture and play out,
log/capture from tape sources, manual and automated
capture from live sources and print to tape/play out
functions.
There are five components to consider when choosing
the Pipeline host system.
• CPU cores and speed (see System Recommenda-
tion section)
• Networks connection (see Pipeline Network
section)
• Disk storage systems (see Pipeline Media Storage
section)
• System RAM (see System Recommendation
section)
• System Video Card
Use the system descriptions, recommended systems
chart, network and storage sections to configure your
Pipeline host system.
Performance tuning of the Pipeline host systems
Perform the following adjustments on the Pipeline host
server to assure proper performance (consult with your
System Administrator or a Telestream Field Sales
Engineer)
Windows systems only:
• From within Performance Options set Processor
scheduling to – Adjust for best performance of:
Programs
• Disable Network throttling (see http://support.
microsoft.com/kb/948066)
• Set value to: FFFFFFFF
• Turn on Data Execution Prevention (DEP) for
essential Windows programs and services only.
• Disable User Account Control (UAC).
• Set the Pipeline host server’s Power Options to
High Performance with all critical functions to be
always on including the disabling of any disk sleep
modes.
• Disable any real-time software applications and
processes that could adversely affect disk I/O
performance or use excessive CPU. In particular
real-time virus scanning of media files as they are
being captured, automatic software updates that
can preempt real-time services, and real-time file
indexing that could consume excessive CPU and
disk I/O.
• Disable all unnecessary Windows services from
Administrative Tools->Services console (consult
your System Administrator)
• Firewalls and packet filtering may result in unpre-
dictable performance and should be avoided
• Use only Intel based enterprise-class Ethernet
Server Network Adapters with Advanced Network
Services (ANS) installed
2
Pipeline
System Setup

Pipeline
System Setup
• For Pipeline channels that are directly connected to
the Pipeline host server’s network adapters,
disable all Networking Property components
except Internet Protocol Version 4 (TCP/IPv4)
• Turn off all power saving options on the Network
Adapters used for Pipeline traffic
• Set Network Adapter transmit and receive buffers
to 1024
• Use static IP Addresses for all Pipeline channels
• Disable any screensaver functions.
Pipeline network
For best results Pipeline device hardware should be
connected directly to a Pipeline host computer or via a
single enterprise class, fully non-blocking professional
quality Ethernet switch via CAT6 Ethernet cabling.
Multiple hops via routers and switches increase the risk
that video frames will not be processed in a timely
manner resulting in lost frames. While not absolutely
required it is highly recommended that the Pipeline
host’s network adapter be based on an Intel Gigabit
Ethernet controller (82571EB, 82546GB, 82576EB,
I340, I350). Only adapters using Intel controllers have
been fully tested with multiple Pipeline channel net-
works. Non-Intel Ethernet adapters embedded on the
Pipeline host workstation or server’s motherboard
should not be used for Pipeline traffic.
Approved Network adapters for Pipeline HD Dual
Intel based enterprise-class Ethernet Network Server
Adapters with Advanced Network Services (ANS)
installed.
• Pro/1000 Series (PT, PF, MT, MF, GT, ET or EF)
• I340 or I350 Series
Capturing across multiple switches is possible but
should not be done in critical Pipeline production
workflows. All switches used for Pipeline traffic must
meet the minimum requirements. Always consult with
your network administrator when attempting to utilize
Pipeline across multiple switches, networks or VLAN’s.
The use of STATIC IP addresses for each Pipeline
channel is required.
• It is required that a fully non-blocking managed
Layer 2/3 Enterprise Edge Ethernet switch with
adequate per-port data buffering be used. The
following switches are highly recommended for
Pipeline systems:
• Extreme Networks Summit X350, X450 and
X650, Dell Force10 S60, Arista 7048T
• Use of a private Pipeline video VLAN is highly
recommended
Special care must be taken to insure that a real-time
process can be achieved. Other network processes,
such as file copying, can consume unbounded band-
width and excessive network stack cycles which can
cause Pipeline real-time streams to be interrupted.
When constructing a Pipeline system utilizing an
existing information network, be aware that any local
processes using the network could affect the Pipeline
traffic. Best practice on the Pipeline host system is to
separate Pipeline traffic from data traffic by using more
than one network adapter within the Pipeline host
computer. Design your system with one (or more) Ether-
net ports dedicated to the incoming Pipeline streams
and separate Ethernet port(s) for general data and
media delivery traffic. As a rule of thumb never exceed
50% usage of any one Ethernet port or network switch.
For example when capturing Pro Res 422 HQ at
220Mbps use one GigE Ethernet port for every two (2)
streams. Multiple port Ethernet cards are required for
this purpose.
There are three basic ways to configure a Pipeline
network setup; direct attached, network attached, and
network attached with LACP (port bonding). Below are
simple examples of all three.
Note: With all Ethernet interfaces, especially add in
cards with more than 2 ports, the physical order of
the interface ports may or may not match the order
they are presented to the user in the O/S. ALWAYS
start by connecting one Pipeline channel at a time to
a single port until you are certain which physical
interface is which in your operating system. For
‘bonded config’ (LACP) both switch and host
machine must support LACP (port failover should not
be enabled)
Physical Port D C B A
O/S Port en2 en5
3

Pipeline Simple Network Example
Pipeline LACP (bonded ports) Network Example
Pipeline Direct Attached Example
Warning: When directly attaching multiple Pipeline channels to the host’s network adapter always connect
them one at a time and configure each on their own subnet (as shown below).
4
Pipeline
System Setup

Pipeline
System Setup
Pipeline media Storage
When deciding on storage, there are two main factors
to consider - the overall sustained read/write perfor-
mance required and whether local data disk buffering
will be required.
To ensure performance and quality, the disk storage
system(s) used with the Pipeline host computer must
be able to meet the demands of reading and writing
one or more real-time SD and HD media streams.
There are many disk system setups that can provide
this level of performance.
For SD, we recommend that the Pipeline host’s disk
storage system provide and maintain a sustained 10
MB/sec transfer rate per stream from the Pipeline host
computer to the media disk (read/write).
For DVCPro HD, ProRes 422 and standard quality
DNxHD, we recommend that the Pipeline host’s disk
storage system provide and maintain a sustained 40
MB/sec transfer rate per stream from the Pipeline host
computer to media disk (read/write) for each stream.
For ProRes 422 HQ and high quality DNxHD, we
recommend at least a 50 MB/sec transfer rate for each
stream.
To evaluate your needs, you must first take into account
the data rate for the format that you’ll be using and the
number of streams your system will be capturing
concurrently. The table below provides approximate
data rates for a variety of formats and resolutions. Use
these tables to calculate the overall read/write perfor-
mance requirements for your main storage disk system
and if needed for your local data buffering storage disk
system.
Using a disk benchmarking tool as ATTO’s Bench32 for
Windows and DiskFire for OSX make sure the media
storage system can accommodate the total bandwidth
necessary, both read and write.
Capture only, writing to internal or direct attached
RAID array: For systems utilizing internal or direct
attached storage for capture only operations you need
only calculate the overall write speed requirements.
Configure your media storage array based on the write
performance needed for the number of streams being
captured.
Four (4) channel Pipeline HD with direct attached Fibre RAID storage
5

Pipeline
System Setup
Capture with concurrent edit or transcode:
For systems that are capturing with concurrent edit or
transcode operations you will need to calculate the
overall read/write speed requirements based on the
write processes of Pipeline streams and read processes
for the edit and transcode operations. To guarantee
error free operations you must use Pipeline’s data
buffering feature utilizing an internal or direct attached
non-shared storage RAID array for data buffering that
meets the desired read/write performance requirement
(see Disk Buffering section below). The Pipeline’s media
storage location (NAS, direct attached shared storage
or shared network folder) must also meet these
minimum read/write performance requirements. When
using NAS, direct attached shared storage or shared
network folder, be sure to verify that your system has
the communications bandwidth for writing all Pipeline
media data to the media storage location in real-time.
Capture writing to NAS or shared network folder:
For systems that are capturing media that is being
written to a NAS or shared network folder, it is required
to use the Pipeline data buffering feature (see Disk
Buffering section below). Data Buffering requires a
separate data buffer RAID array to guarantee error free
write operations of Pipeline media streams. Calculating
the disk buffer RAID array read/write performance
requirements using 2X the required Pipeline stream’s
write requirements. This is needed to support a write
and a read operation on the data buffer array for each
Pipeline stream. The Pipeline’s media storage location
(NAS, direct attached shared storage or shared network
folder) must meet the minimum write performance
requirements. When using NAS or shared network
folder, be sure to verify that your system has the
communications bandwidth for writing all Pipeline
media data to the media storage location in real-time.
Four (4) channel Pipeline HD with direct attached data buffer RAID
Caution: Capturing video is a very disk intensive operation. It is extremely important that you create a system that has
adequate disk performance for writing and reading real-time video files. In addition to the information provided in this
guide make sure that the disk system used for Pipeline operations are defragmented often and that any disk sleep/
stand-by power settings are disabled. Fragmented drives or drives in sleep/stand-by states can dramatically decrease
performance and may cause capture or play out operations to fail.
6

Pipeline
System Setup
Example Systems
Distributed Pipeline model – individual workstations
controlling individual channels. Each workstation
meeting the single Pipeline channel spec with a
network to support all traffic.
Centralized Pipeline ingest model – A central worksta-
tion controlling multiple Pipeline channels. The central
workstation meets the multiple Pipeline channel spec
with a network and storage system to support the
workflow.
Four (4) channel HD with data buffering – Four (4)
channels of HD hosted by a central workstation with
delivery to a NAS. Data Buffering is enabled through
directly attached RAID storage.
Multiple channel Transcode while capture – Multiple
Pipeline channels hosted by a central workstation with
delivery to a directly attached storage device. The
storage is shared to the transcoding servers for
concurrent processing.
Multiple channel edit while capture – Multiple Pipeline
channels hosted by a central workstation with delivery
to shared storage device. Internal 15K RAID array is
used as a Disk Buffer in case the shared storage
cannot maintain real time writes.
7

Pipeline
System Setup
Using a Pipeline HD Dual and Vantage on a single system
Pipeline HD Dual and Vantage can be installed on the
same server in order to create a two (2) channel
baseband ingest solution directly integrated with
Vantage. This will allow for real-time or near real-time
file creation into virtually any format from live or tape
sources.
Generally Pipeline Control processes do not use a great
deal of CPU resources allowing Vantage workflow
processes to utilize the majority of available CPU
resources. However close attention must be taken to
assure that the Vantage server can meet the Pipeline’s
CPU, network bandwidth and disk I/O requirements.
Follow the Pipeline configuration and setup procedures
as described in this document in order to determine the
host system requirements for an integrated two (2)
channel Pipeline/Vantage server.
There are five components to consider when choosing
a two (2) channel Pipeline/Vantage host system.
• CPU cores and speed (see System Recommenda-
tion section)
• Faster is always better
• Network connections (see Pipeline Network
section)
• Separating Pipeline channel traffic from general
data traffic by directly attaching each Pipeline
channel is required
• Media Disk storage systems (see Pipeline Media
Storage section)
• Disk I/O fast enough to support real-time
Pipeline file writing, Vantage file writing and
Vantage file reading
• System RAM (see System Recommendation
section)
• Enough RAM to support all Pipeline and Vantage
processes
• 8GB per channel is the general rule
• More is always better
• System Video Card
Use the system descriptions, recommended systems
chart, network and storage sections in this document to
configure your Pipeline/Vantage host system.
Pipeline Host computer recommendations
Network and Hard Disk Performance Requirements (per stream)
CODEC
Disk read/write
(MB/s)
Network throughput
(Mb/s)
Storage required
with 8 channels of
audio
SD Codecs
Motion JPEG A 0.75 8 5GB/hr
DV/DVCPro 4.4 35 15GB/hr
IMX30 5.3 43 16GB/hr
IMX40 7 56 20GB/hr
IMX50/DVCPro50 8.3 67 25GB/hr
ProRes 422 7. 3 58 21GB/hr
Uncompressed 8bit
422
22 175 80GB/hr
HD Codecs (range
based on codec and
bit rate)
DVCPro HD 16.5 135 73GB/hr
ProRes 422 HQ 10-36 62-240 50-100GB/hr
8

Pipeline
System Setup
www.telestream.net | [email protected] |tel [phone]
Specifications subject to change without notice. Copyright © 2014 Telestream, Inc.
Telestream, CaptionMaker, Episode, Flip4Mac, FlipFactory, Flip Player, Lightspeed,
ScreenFlow, Vantage, Wirecast, GraphicsFactory, MetaFlip, MotionResolve, and
Split-and-Stitch are registered trademarks and Pipeline, MacCaption, e-Captioning,
and Switch are trademarks of Telestream, Inc. All other trademarks are the property of
their respective owners.
June 2014
Systems Recommendations1
General guidelines
• CPU
• Single or Dual channel systems - Dual Dual
Core 2.3Ghz CPU or faster
• Three or Four channel systems - Dual Quad
Core 2.3Ghz CPU or faster
• Four or Five channel - Dual Six Core 2.3Ghz
CPU or faster
• RAM - 4GB RAM per channel
• Operating System – Check Telestream web site
for the latest supported OS
• Media Storage sustained read and/or write speeds
(separate from OS drive)
• Capture only – 40MB/s per channel
• Play out only – 60MB/s per channel
• Transcode/Edit/Disk Buffering while Capture –
100MB/s per channel
1
All systems assume media storage local (internal or
direct attached) to the host system with adequate
storage to support 24 hours of content at the chosen
format, and drive array is less than 80% full without
serious fragmentation.
Edit while ingest and transcode while ingest workflows
require increased disk I/O performance to support the
additional read processes, and may require disk buffer
cache drive system (see Pipeline User Guide disk
buffering details).
Viewing HD preview stream(s) can consumes high
levels of CPU which may cause dropped frames. It is
recommended that preview be turned off during
capture.
9