The tutorial shows you how to integrate das element with your render farm.
In this example Deadline with Linux is used. The concept however applies to every other render farm manager as well.
Watch this video tutorial to get started!
List of useful tags
The keys get resolved during ingest/proxy generation time and will be replaced with the actual value.
You can use the post render hook to add custom values.
key | Description | Example value |
|---|---|---|
| Custom additional values from the user that get returned from a post render hook script. (post_render.py) |
|
| Pixel Width from either the source file, database entity, the transcoding preset (like thumbnail) or the overwrite from the Path Values |
|
| Pixel Height from either the source file, database entity, the transcoding preset (like thumbnail) or the overwrite from the Path Values |
|
| Pixel Aspect Ratio from either the source file, the transcoding preset (like thumbnail) or the overwrite from the Path Values |
|
| First frame from either the source file, database entity or the overwrite from the Path Values |
|
| Last frame from either the source file, database entity or the overwrite from the Path Values |
|
| Name of the element in the database |
|
| Database ID of the element |
|
| The type of media which is either its |
|
| File path to element |
|
| File path to elements filmstrip |
|
| File path to elements proxy movie file |
|
| File path to elements thumbnail |
|
| File path to original source |
|
| Full file name of original source file(s) |
|
| Basename file name, without extension and frame counter, of original source file(s) |
|
| File extension of original source file(s) |
|
| Colorspace of the original source file(s) |
|
Python hooks
For each transcoding task, the Python hook for ‘pre_render’ (pre_render.py) and ‘post render' (post_render.py) gets the Job ID from the render farm. If the main task isn’t finished then there is nothing to render for the proxy tasks and they will probably fail.
It is why it is important to set a dependencies on the proxy tasks to the main task.
Pre Render Hook
The input is the resolver data dictionary which will be used to resolve the path pattern. You can modify it here before the transcoding task gets processed.
You will need to return the same dictionary with your changes included.
In the example below, in order to resolve a path pattern like <custom.dependecy> you need to add some value for the custom data. If that is not provided the resolve would otherwise fail if you re-render the proxies and there is no main task to provide the dependency which we normally would get from the post render hook after sending the task to the render farm.
# Example file if you use some custom dependency in your submission scripts.
# To resolve a path pattern like <custom.dependecy> you need to add some value for the custom data.
# If that is not provided the resolve would otherwise fail if you re-render the proxies and there is no main task to provide the depdency.
import sys
def main(*args):
data = args[0]
# make sure to set some value for the dependency.
# otherwise it will failed when resolving the path pattern '<custom.dependency>'
if not data['template_values'].get('custom'):
data['template_values']['custom'] = {'dependency': ''}
return data
if __name__ == '__main__':
main(sys.argv[1:])
Post Render Hook
The input is the output of the process called by the custom command task.
You will have to return a Dictionary which will be added as custom to the resolver data and can later be accessed in the Path Builder. return {'dependency': job_id} can later be resolved with <custom.dependency>
# Example Python script for post render hook (post_render.py):
import sys
import re
def main(*args):
# args[0] is the output from the process calling the 'exec' and 'params'
job_output = args[0]
job_id = ''
if job_output:
match = re.search(r'JobID=([a-zA-Z0-9]*)', job_output)
if match:
job_id = match.group(1)
print('Job ID: {}'.format(job_id))
# returns data that can later be access as "custom"
# in this example: <custom.dependency>
return {'dependency': job_id}
if __name__ == '__main__':
main(sys.argv[1:])
Transcoding tasks
Transcoding task: main element
Simple command line
# exec: # Linux: cp # Windows: copy params: \"<source.path>\" \"<path>\"
Python script
The source file(s) can be a single file, like a single frame or a movie file or a sequence of files.
That makes stuff a little bit more complicated if you want to have a generic file to deal with all these cases.
Find below an example script which was tested on Linux and Windows with Deadline.
Please make sure to install the fileseq Python package. 
Thank you to the developers!
"""
__ __ __
____/ /___ ______ ___ / /__ ____ ___ ___ ____ / /_
/ __ / __ `/ ___/ / _ \/ / _ \/ __ `__ \/ _ \/ __ \/ __/
/ /_/ / /_/ (__ ) / __/ / __/ / / / / / __/ / / / /_
\__,_/\__,_/____/ \___/_/\___/_/ /_/ /_/\___/_/ /_/\__/
Create a transcoding template with a 'custom command' task.
This example is for Linux and Deadline.
Requirements:
The environment variable 'DASELEMENT_RESOURCES' needs to point to some network
share where this script is located. In this example to copy a image sequence you
need the package fileseq (https://pypi.org/project/Fileseq/)
$ pip install fileseq
######## Linux ########
exec:
/opt/Thinkbox/Deadline10/bin/deadlinecommand
params:
-SubmitCommandLineJob -startupdirectory "$DASELEMENT_RESOURCES" -executable "/usr/bin/python" -arguments "scripts/custom/copy_main.py \"<source.path>\" \"<path>\" <media_type> <frame_first> <frame_last>" -frames 1 -chunksize 1 -priority 50 -name "[das element] <name> - main" -prop BatchName="[das element] <name>"
######## Windows ########
exec:
"C:/Program Files/Thinkbox/Deadline10/bin/deadlinecommand.exe"
params:
-SubmitCommandLineJob -startupdirectory "%DASELEMENT_RESOURCES%" -executable "C:/Python/Python37/python.exe" -arguments "scripts/custom/copy_main.py \"<source.path>\" \"<path>\" <media_type> <frame_first> <frame_last>" -frames 1 -chunksize 1 -priority 50 -name "[das element] <name> - main" -prop BatchName="[das element] <name>"
"""
import sys
import shutil
from fileseq import FileSequence
from pathlib import Path
def copy_file_sequence(source, output, frame_first, frame_last):
# get all source file paths
source_files = [Path(path) for path in list(FileSequence(source))]
# get all element file paths
sequence = FileSequence(output.replace('#', '@'))
sequence.setFrameRange('{}-{}'.format(frame_first, frame_last))
output_files = [Path(path) for path in list(sequence)]
# create destination folder if it does not yet exist
if not Path(output).parent.exists():
Path(output).parent.mkdir()
# copy files
for item in zip(source_files, output_files):
if not item[0].exists():
print('File does not exist: {}'.format(item[0]))
print('Failed to copy file: {}'.format(item[1]))
continue
shutil.copy2(str(item[0]), str(item[1]))
def main(*args):
path_source, path_output, media_type, frame_first, frame_last = args[0]
if media_type == 'sequence':
copy_file_sequence(path_source, path_output, frame_first, frame_last)
return True
shutil.copy2(path_source, path_output)
return True
if __name__ == '__main__':
main(sys.argv[1:])
Transcoding task: thumbnail
In this example we use FFmpeg to create the thumbnail.
Simple command line
It shows you a way how to do it without any custom scripting. However this might be not very practical in production, be cause e need to create two transcoding tasks. One for single image files and another for movies/sequences. They have different command lines because movies/sequences need to know which frame is the thumbnail.
# exec: /opt/Thinkbox/Deadline10/bin/deadlinecommand # params: # command line for single images -SubmitCommandLineJob -executable "/usr/bin/ffmpeg" -arguments "-i "<path>" -y -vframes 1 -an -s <width>x<height> "<path_thumbnail>"" -frames 1 -chunksize 1 -priority 50 -name "[das element] <name> - thumbnail image" -prop BatchName="[das element] <name>" -prop JobDependencies="<custom.dependency>" # command line for movie files/sequences (the additional time flag (-ss) is defined) -SubmitCommandLineJob -executable "/usr/bin/ffmpeg" -arguments "-i "<path>" -y -vframes 1 -an -s <width>x<height> -ss <frame_first> "<path_thumbnail>"" -frames 1 -chunksize 1 -priority 50 -name "[das element] <name> - thumbnail movie/sequence" -prop BatchName="[das element] <name>" -prop JobDependencies="<custom.dependency>"
Python script
Creating proxy thumbnail can also be done with a custom python script. FFmpeg is used to render the proxy files in this example.
There is a parameter that describes the type of media (<media_type>) this is either image, movie or sequence. This helps you to determine how to deal with the different file paths types. It can be included in the Custom Commands Parameters.
"""
__ __ __
____/ /___ ______ ___ / /__ ____ ___ ___ ____ / /_
/ __ / __ `/ ___/ / _ \/ / _ \/ __ `__ \/ _ \/ __ \/ __/
/ /_/ / /_/ (__ ) / __/ / __/ / / / / / __/ / / / /_
\__,_/\__,_/____/ \___/_/\___/_/ /_/ /_/\___/_/ /_/\__/
Create a transcoding template with a 'custom command' task.
This example is for Linux. For Windows (examples below) adjust the ffmpeg executable paths.
Requirements:
The environment variable 'DASELEMENT_RESOURCES' needs to point to some network
share where this script is located.
######## Linux ########
exec:
/opt/Thinkbox/Deadline10/bin/deadlinecommand
params:
-SubmitCommandLineJob -startupdirectory "$DASELEMENT_RESOURCES" -executable "/usr/bin/python" -arguments "scripts/custom/create_thumbnail.py \"<path>\" \"<path_thumbnail>\" <width> <height> <media_type> <frame_first> <frame_last>" -name "[das element] <name> - thumbnail" -prop BatchName="[das element] <name>" -frames 1 -chunksize 1 -priority 50 -prop JobDependencies="<custom.dependency>"
######## Windows ########
exec:
"C:/Program Files/Thinkbox/Deadline10/bin/deadlinecommand.exe"
params:
-SubmitCommandLineJob -startupdirectory "%DASELEMENT_RESOURCES%" -executable "C:/Python/Python37/python.exe" -arguments "scripts/custom/create_thumbnail.py \"<path>\" \"<path_thumbnail>\" <width> <height> <media_type> <frame_first> <frame_last>" -name "[das element] <name> - thumbnail" -prop BatchName="[das element] <name>" -frames 1 -chunksize 1 -priority 50 -prop JobDependencies="<custom.dependency>"
"""
import os
import re
import sys
import subprocess
from pathlib import Path
######## Linux ########
EXECUTABLE_FFMPEG = '/usr/bin/ffmpeg'
EXECUTABLE_FFPROBE = '/usr/bin/ffprobe'
######## Windows ########
# EXECUTABLE_FFMPEG = 'C:/ffmpeg/bin/ffmpeg.exe'
# EXECUTABLE_FFPROBE = 'C:/ffmpeg/bin/ffprobe.exe'
def frames_to_timecode(frames, frame_rate):
h = int(frames / 86400)
m = int(frames / 1440) % 60
s = int((frames % 1440) / frame_rate)
f = frames % 1440 % frame_rate
return '%02d:%02d:%02d.%02d' % (h, m, s, f)
def get_movie_frame_rate(path):
command = [
EXECUTABLE_FFPROBE, '-v', '0', '-of', 'csv=p=0', '-select_streams',
'v:0', '-show_entries', 'stream=r_frame_rate', '"{}"'.format(path)
]
command_string = ' '.join(command)
return eval(os.popen(command_string).read())
def main(*args):
path, path_output, width, height, media_type, first, last = args[0]
frame_center = round((int(last) - int(first)) / 2)
executable = [EXECUTABLE_FFMPEG]
arguments = []
if media_type == 'image':
arguments += ['-i', path]
if media_type == 'movie':
frame_rate = get_movie_frame_rate(path)
timestamp = frames_to_timecode(frame_center, frame_rate)
arguments += ['-i', path, '-ss', timestamp]
if media_type == 'sequence':
frame_rate = 24 # Set random frame rate. Works fine for image sequences
timestamp = frames_to_timecode(frame_center, frame_rate)
path_string_format = re.sub(r"[#]+", "%d", str(path))
arguments += [
'-start_number',
str(first), '-r',
str(frame_rate), '-f', 'image2', '-i', path_string_format
]
arguments += ['-ss', timestamp]
filter_scale = 'scale={}:{}:'.format(width, height)
filter_scale += 'force_original_aspect_ratio=decrease,'
filter_scale += 'pad={}:{}:(ow-iw)/2:(oh-ih)/2'.format(width, height)
arguments += [
'-y', '-vf', filter_scale, '-vcodec', 'png', '-q:v', '5', '-frames:v',
'1', path_output
]
command = executable + arguments
print('Command to execute:')
print(' '.join(command))
if not Path(path_output).parent.exists():
Path(path_output).parent.mkdir()
process = subprocess.Popen(command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
output, error = process.communicate()
if process.returncode != 0:
print('Something went wrong!')
print(process.returncode)
print(output)
print(error)
return process.returncode
if __name__ == '__main__':
main(sys.argv[1:])
Transcoding task: proxy movie
In this example we use FFmpeg to create proxy movie file.
Simple command line
# exec: /opt/Thinkbox/Deadline10/bin/deadlinecommand # params: -SubmitCommandLineJob -executable "/usr/bin/ffmpeg" -arguments "-i \"<path>\" -y -r 24 -vf \"scale=<width>:<height>:force_original_aspect_ratio=decrease,pad=<width>:<height>:(ow-iw)/2:(oh-ih)/2\" -vcodec libx264 -crf 23 -preset faster -tune film -pix_fmt yuv420p -framerate 24 -timecode 00:00:41:17 -acodec copy \"<path_proxy>\"" -frames 1 -chunksize 1 -priority 50 -name "[das element] <name> - proxy movie" -prop BatchName="[das element] <name>" -prop JobDependencies="<custom.dependency>"
Python script
Creating proxy movie files can also be done with a custom python script. FFmpeg is used to render the proxy files in this example.
There is a parameter that describes the type of media (<media_type>) this is either image, movie or sequence. This helps you to determine how to deal with the different file paths types. It can be included in the Custom Commands Parameters.
"""
__ __ __
____/ /___ ______ ___ / /__ ____ ___ ___ ____ / /_
/ __ / __ `/ ___/ / _ \/ / _ \/ __ `__ \/ _ \/ __ \/ __/
/ /_/ / /_/ (__ ) / __/ / __/ / / / / / __/ / / / /_
\__,_/\__,_/____/ \___/_/\___/_/ /_/ /_/\___/_/ /_/\__/
Create a transcoding template with a 'custom command' task.
This example is for Linux. For Windows (examples below) adjust the ffmpeg executable paths.
Requirements:
The environment variable 'DASELEMENT_RESOURCES' needs to point to some network
share where this script is located.
######## Linux ########
exec:
/opt/Thinkbox/Deadline10/bin/deadlinecommand
params:
-SubmitCommandLineJob -startupdirectory "$DASELEMENT_RESOURCES" -executable "/usr/bin/python" -arguments "scripts/custom/create_proxy.py \"<path>\" \"<path_proxy>\" <width> <height> <media_type> <frame_first>" -name "[das element] <name> - proxy" -prop BatchName="[das element] <name>" -frames 1 -chunksize 1 -priority 50 -prop JobDependencies="<custom.dependency>"
######## Windows ########
exec:
"C:/Program Files/Thinkbox/Deadline10/bin/deadlinecommand.exe"
params:
-SubmitCommandLineJob -startupdirectory "%DASELEMENT_RESOURCES%" -executable "C:/Python/Python37/python.exe" -arguments "scripts/custom/create_proxy.py \"<path>\" \"<path_proxy>\" <width> <height> <media_type> <frame_first>" -name "[das element] <name> - proxy" -prop BatchName="[das element] <name>" -frames 1 -chunksize 1 -priority 50 -prop JobDependencies="<custom.dependency>"
"""
import os
import re
import sys
import subprocess
from pathlib import Path
######## Linux ########
EXECUTABLE_FFMPEG = '/usr/bin/ffmpeg'
EXECUTABLE_FFPROBE = '/usr/bin/ffprobe'
######## Windows ########
# EXECUTABLE_FFMPEG = 'C:/ffmpeg/bin/ffmpeg.exe'
# EXECUTABLE_FFPROBE = 'C:/ffmpeg/bin/ffprobe.exe'
def frames_to_timecode(frames, frame_rate):
h = int(frames / 86400)
m = int(frames / 1440) % 60
s = int((frames % 1440) / frame_rate)
f = frames % 1440 % frame_rate
return '%02d:%02d:%02d.%02d' % (h, m, s, f)
def get_movie_frame_rate(path):
command = [
EXECUTABLE_FFPROBE, '-v', '0', '-of', 'csv=p=0', '-select_streams',
'v:0', '-show_entries', 'stream=r_frame_rate', '"{}"'.format(path)
]
command_string = ' '.join(command)
return eval(os.popen(command_string).read())
def main(*args):
path, path_output, width, height, media_type, frame_first = args[0]
frame_rate = 24 # Set random frame rate. Works for image sequences but is not accurate for movie files!
executable = [EXECUTABLE_FFMPEG]
arguments = []
if media_type == 'image':
print('No sequence or movie file - will not render filmstrip')
return False
# as an alternative you could loop an still image for 1 second
# arguments += ['-loop', '1', '-i', path, '-t', '1']
if media_type == 'movie':
frame_rate = get_movie_frame_rate(path)
arguments += ['-i', path]
if media_type == 'sequence':
path_string_format = re.sub(r"[#]+", "%d", str(path))
arguments += [
'-start_number',
str(frame_first), '-r',
str(frame_rate), '-f', 'image2', '-i', path_string_format
]
timestamp_start = frames_to_timecode(int(frame_first), frame_rate)
arguments += [
'-y', '-r',
str(frame_rate), '-vf',
'scale={0}:{1}:force_original_aspect_ratio=decrease,pad={0}:{1}:(ow-iw)/2:(oh-ih)/2'
.format(width, height), '-vcodec', 'libx264', '-crf', '23', '-preset',
'faster', '-tune', 'film', '-pix_fmt', 'yuv420p', '-framerate',
str(frame_rate), '-timecode', timestamp_start, '-acodec', 'copy',
path_output
]
command = executable + arguments
print('Command to execute:')
print(' '.join(command))
if not Path(path_output).parent.exists():
Path(path_output).parent.mkdir()
process = subprocess.Popen(command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
output, error = process.communicate()
if process.returncode != 0:
print('Something went wrong!')
print(process.returncode)
print(output)
print(error)
return process.returncode
if __name__ == '__main__':
main(sys.argv[1:])
qweqwe
Transcoding task: filmstrip
In this example we use FFmpeg to create a filmstrip for movie files and sequences of images.
Python script
This can be done with a custom python script. FFmpeg is used to render the filmstrip in this example.
There is a parameter that describes the type of media (<media_type>) this is either image, movie or sequence. This helps you to determine how to deal with the different file paths types. It can be included in the Custom Commands Parameters.
"""
__ __ __
____/ /___ ______ ___ / /__ ____ ___ ___ ____ / /_
/ __ / __ `/ ___/ / _ \/ / _ \/ __ `__ \/ _ \/ __ \/ __/
/ /_/ / /_/ (__ ) / __/ / __/ / / / / / __/ / / / /_
\__,_/\__,_/____/ \___/_/\___/_/ /_/ /_/\___/_/ /_/\__/
Create a transcoding template with a 'custom arguments' task.
This example is for Linux. For Windows (examples below) adjust the ffmpeg executable paths.
Requirements:
The environment variable 'DASELEMENT_RESOURCES' needs to point to some network
share where this script is located.
######## Linux ########
exec:
/opt/Thinkbox/Deadline10/bin/deadlinecommand
params:
-SubmitCommandLineJob -startupdirectory "$DASELEMENT_RESOURCES" -executable "/usr/bin/python" -arguments "scripts/custom/create_filmstrip.py \"<path>\" \"<path_filmstrip>\" <height> <media_type> <frame_first> <frame_last>" -name "[das element] <name> - filmstrip" -prop BatchName="[das element] <name>" -frames 1 -chunksize 1 -priority 50 -prop JobDependencies="<custom.dependency>"
######## Windows ########
exec:
"C:/Program Files/Thinkbox/Deadline10/bin/deadlinecommand.exe"
params:
-SubmitCommandLineJob -startupdirectory "%DASELEMENT_RESOURCES%" -executable "C:/Python/Python37/python.exe" -arguments "scripts/custom/create_filmstrip.py \"<path>\" \"<path_filmstrip>\" <height> <media_type> <frame_first> <frame_last>" -name "[das element] <name> - filmstrip" -prop BatchName="[das element] <name>" -frames 1 -chunksize 1 -priority 50 -prop JobDependencies="<custom.dependency>"
"""
import os
import re
import sys
from pathlib import Path
######## Linux ########
EXECUTABLE_FFMPEG = '/usr/bin/ffmpeg'
EXECUTABLE_FFPROBE = '/usr/bin/ffprobe'
######## Windows ########
# EXECUTABLE_FFMPEG = 'C:/ffmpeg/bin/ffmpeg.exe'
# EXECUTABLE_FFPROBE = 'C:/ffmpeg/bin/ffprobe.exe'
def frames_to_timecode(frames, frame_rate):
h = int(frames / 86400)
m = int(frames / 1440) % 60
s = int((frames % 1440) / frame_rate)
f = frames % 1440 % frame_rate
return '%02d:%02d:%02d.%02d' % (h, m, s, f)
def get_movie_frame_rate(path):
command = [
EXECUTABLE_FFPROBE, '-v', '0', '-of', 'csv=p=0', '-select_streams',
'v:0', '-show_entries', 'stream=r_frame_rate', '"{}"'.format(path)
]
command_string = ' '.join(command)
result = eval(os.popen(command_string).read())
print('Movie frame rate: {}'.format(result))
return result
def get_frame_numbers(frame_first, frame_last, number_of_frames):
frame_numbers = [] # the frame numbers used for the filmstrip
frame_numbers_total = int(frame_last) - int(frame_first) + 1
# calculate which frames to use for the filmstrip
# the frames will be evenly distributed across the total length of the element
frame_mod = float(frame_numbers_total) / float(number_of_frames)
if frame_mod < 1.0:
frame_mod = 1.0
for num in range(frame_numbers_total):
if float(num) % frame_mod >= 1.0:
continue
frame_numbers.append(int(num))
print('Frame numbers:')
print(frame_numbers[:number_of_frames])
# limit number of frames to the given frames value.
# This fixes potential some rounding calculation errors
return frame_numbers[:number_of_frames]
def main(*args):
path, path_output, height, media_type, frame_first, frame_last = args[0]
if media_type == 'image':
print('No sequence or movie file - will not render filmstrip')
return False
executable = [EXECUTABLE_FFMPEG]
arguments = ['-y', '-vsync', '0']
streams = ''
streams_ids = ''
number_of_frames = 24 # the number of frames that the filmstrip has
frame_width = int((16. / 9.) * float(height)) # image has 16:9 image ratio
frame_numbers = get_frame_numbers(frame_first, frame_last,
number_of_frames)
if media_type == 'movie':
frame_rate = get_movie_frame_rate(path)
else:
frame_rate = 24 # set some random frame rate for the image sequence
for stream_number, frame_number in enumerate(frame_numbers):
if media_type == 'sequence':
# replace all hashtag (#) with %d
path_string_format = re.sub(r'[#]+', '%d', str(path))
arguments += [
'-ss',
frames_to_timecode(int(frame_number), frame_rate), '-t',
'0.0001', '-noaccurate_seek', '-start_number',
str(frame_first), '-r',
str(frame_rate), '-f', 'image2', '-i',
'"{}"'.format(path_string_format)
]
else:
arguments += [
'-ss',
frames_to_timecode(int(frame_number), frame_rate), '-t',
'0.0001', '-noaccurate_seek', '-i', '"{}"'.format(path)
]
# scale down each frame from the input
streams += '[{0}:v]scale={1}:{2}:force_original_aspect_ratio=decrease,pad={1}:{2}:(ow-iw)/2:(oh-ih)/2[{0}];'.format(
stream_number, frame_width, height)
streams_ids += '[{}]'.format(stream_number)
streams += '{}hstack=inputs={},pad={}:{}:0:0[out]'.format(
streams_ids, number_of_frames, number_of_frames * frame_width, height)
arguments += [
'-filter_complex', '"{}"'.format(streams), '-map', '[out]',
'-frames:v', '1', '"{}"'.format(path_output)
]
command = executable + arguments
print('arguments to execute:')
print(' '.join(command))
if not Path(path_output).parent.exists():
Path(path_output).parent.mkdir()
# need to use os.popen here because subprocess.Popen has some limitations
# on the maximum characters for the command
stream = os.popen(' '.join(command))
output = stream.read()
print(output)
if __name__ == '__main__':
main(sys.argv[1:])