Connect to the renderfarm

Owned by Jonas Kluger
Last updated: Aug 31, 2023
5 min read

The tutorial shows you how to integrate das element with your render farm.
In this example AWS Deadline is used. The concept however applies to every other render farm manager as well.

ย 

Watch this video tutorial to get started!

Follow these steps

  • create shared resources folder on the server

    • add environment variable DASELEMENT_RESOURCES to point to that shared folder

  • add pre & post render hooks (needed for the job dependencies)

  • adjust transcoding templates

    • copy and paste the "executable" and "parameters" from the example files
      can be found on the top of each Python files

  • add new Path Values for the width and height

Example files

The latest example files for renderfarm jobs can be found here:

GitHub - das-element/resources: Resources and example scripts for the software

ย 

Python hooks

For each transcoding task, the Python hook for โ€˜pre_renderโ€™ (pre_render.py) and โ€˜post render' (post_render.py) gets the Job ID from the render farm. If the main task isnโ€™t finished then there is nothing to render for the proxy tasks and they will probably fail.

It is why it is important to set a dependencies on the proxy tasks to the main task.

Pre Render Hook

The input is the resolver data dictionary which will be used to resolve the path pattern. You can modify it here before the transcoding task gets processed.

You will need to return the same dictionary with your changes included.

In the example below, in order to resolve a path pattern like <custom.dependecy> you need to add some value for the custom data. If that is not provided the resolve would otherwise fail if you re-render the proxies and there is no main task to provide the dependency which we normally would get from the post render hook after sending the task to the render farm.

Check out the example script here:
https://github.com/das-element/resources/blob/main/scripts/hooks/examples/deadline/pre_render.py

Post Render Hook

The input is the output of the process called by the custom command task.

You will have to return a Dictionary which will be added as custom to the resolver data and can later be accessed in the Path Builder. return {'dependency': job_id} can later be resolved with <custom.dependency>

Check out the example script here:
https://github.com/das-element/resources/blob/main/scripts/hooks/examples/deadline/post_render.py

Transcoding tasks

Examples for Nuke

Here are some example scripts and templates on how to use The Foundry Nuke for transcoding elements.

Check out the example script here:
resources/scripts/custom/examples/nuke at main ยท das-element/resources

Transcoding task: main element

Simple command line

# exec: # Linux: cp # Windows: copy params: \"<source.path>\" \"<path>\"

Python script

The source file(s) can be a single file, like a single frame or a movie file or a sequence of files.
That makes stuff a little bit more complicated if you want to have a generic file to deal with all these cases.

Find below an example script which was tested on Linux and Windows with Deadline.
Please make sure to install the fileseq Python package. Thank you to the developers!

Check out the example script here:
https://github.com/das-element/resources/blob/main/scripts/custom/examples/copy_main.py

Transcoding task: thumbnail

In this example we use FFmpeg to create the thumbnail.

Simple command line

It shows you a way how to do it without any custom scripting. However this might be not very practical in production, be cause e need to create two transcoding tasks. One for single image files and another for movies/sequences. They have different command lines because movies/sequences need to know which frame is the thumbnail.

# exec: /opt/Thinkbox/Deadline10/bin/deadlinecommand # params: # command line for single images -SubmitCommandLineJob -executable "/usr/bin/ffmpeg" -arguments "-i "<path>" -y -vframes 1 -an -s <width>x<height> "<path_thumbnail>"" -frames 1 -chunksize 1 -priority 50 -name "[das element] <name> - thumbnail image" -prop BatchName="[das element] <name>" -prop JobDependencies="<custom.dependency>" # command line for movie files/sequences (the additional time flag (-ss) is defined) -SubmitCommandLineJob -executable "/usr/bin/ffmpeg" -arguments "-i "<path>" -y -vframes 1 -an -s <width>x<height> -ss <frame_first> "<path_thumbnail>"" -frames 1 -chunksize 1 -priority 50 -name "[das element] <name> - thumbnail movie/sequence" -prop BatchName="[das element] <name>" -prop JobDependencies="<custom.dependency>"

Python script

Creating proxy thumbnail can also be done with a custom python script. FFmpeg is used to render the proxy files in this example.

There is a parameter that describes the type of media (<media_type>) this is either image, movie or sequence. This helps you to determine how to deal with the different file paths types. It can be included in the Custom Commands Parameters.

Check out the example script here:
https://github.com/das-element/resources/blob/main/scripts/custom/examples/create_thumbnail.py

Transcoding task: proxy movie

In this example we use FFmpeg to create proxy movie file.

Simple command line

# exec: /opt/Thinkbox/Deadline10/bin/deadlinecommand # params: -SubmitCommandLineJob -executable "/usr/bin/ffmpeg" -arguments "-i \"<path>\" -y -r 24 -vf \"scale=<width>:<height>:force_original_aspect_ratio=decrease,pad=<width>:<height>:(ow-iw)/2:(oh-ih)/2\" -vcodec libx264 -crf 23 -preset faster -tune film -pix_fmt yuv420p -framerate 24 -timecode 00:00:41:17 -acodec copy \"<path_proxy>\"" -frames 1 -chunksize 1 -priority 50 -name "[das element] <name> - proxy movie" -prop BatchName="[das element] <name>" -prop JobDependencies="<custom.dependency>"

Python script

Creating proxy movie files can also be done with a custom python script. FFmpeg is used to render the proxy files in this example.

There is a parameter that describes the type of media (<media_type>) this is either image, movie or sequence. This helps you to determine how to deal with the different file paths types. It can be included in the Custom Commands Parameters.

Check out the example script here:
https://github.com/das-element/resources/blob/main/scripts/custom/examples/create_proxy.py

Transcoding task: filmstrip

In this example we use FFmpeg to create a filmstrip for movie files and sequences of images.

Python script

This can be done with a custom python script. FFmpeg is used to render the filmstrip in this example.

There is a parameter that describes the type of media (<media_type>) this is either image, movie or sequence. This helps you to determine how to deal with the different file paths types. It can be included in the Custom Commands Parameters.

Check out the example script here:
https://github.com/das-element/resources/blob/main/scripts/custom/examples/create_filmstrip.py

ย 

List of useful tags

The keys get resolved during ingest/proxy generation time and will be replaced with the actual value.
You can use the post render hook to add custom values.

key

Description

Example value

key

Description

Example value

custom.???

Custom additional values from the user that get returned from a post render hook script. (post_render.py)

{dependency: '611b8af0f369f3140ace450c'}

width

Pixel Width from either the source file, database entity, the transcoding preset (like thumbnail) or the overwrite from the Path Values

1920

height

Pixel Height from either the source file, database entity, the transcoding preset (like thumbnail) or the overwrite from the Path Values

1080

pixel_aspect

Pixel Aspect Ratio from either the source file, the transcoding preset (like thumbnail) or the overwrite from the Path Values

1

frame_first

First frame from either the source file, database entity or the overwrite from the Path Values

1001

frame_last

Last frame from either the source file, database entity or the overwrite from the Path Values

1095

name

Name of the element in the database

artificial_00012

id

Database ID of the element

82

media_type

The type of media which is either its image, movie or sequence

movie

path

File path to element

/some/path/artificial/artificial_00012/main_1920x1080_source/artificial_00012.mov

path_filmstrip

File path to elements filmstrip

/some/path/artificial/artificial_00012/filmstrip_1920x1080_srgb/artificial_00012.jpg

path_proxy

File path to elements proxy movie file

/some/path/artificial/artificial_00012/proxy_1920x1080_srgb/artificial_00012.mov

path_thumbnail

File path to elements thumbnail

/some/path/artificial/artificial_00012/thumb_960x540_srgb/artificial_00012.jpg

path_source
source.path

File path to original source

/source/file/path/folder/the_source_file.mov

source.name

Full file name of original source file(s)

foo_bar.%04.exr

source.basename

Basename file name, without extension and frame counter, of original source file(s)

foo_bar

source.extension

File extension of original source file(s)

exr

colorspace_source

Colorspace of the original source file(s)

linear

ย