[Sidefx-houdini-list] render farms, pipelines and file systems

Drew Whitehouse Drew.Whitehouse at anu.edu.au
Tue Jan 6 22:28:35 EST 2009

Hi all,

I'm interested in hearing about experiences people have had
implementing render farms where you *don't* have a shared file system
between workstations and render farms. I currently have a solution
than involves shipping resources back and forth. However, it requires
parsing IFD files and heuristically determining what resources are
needed for a render and what results needed to be transported back
post render. I'd like to make this more robust and I'm sure that this
is something that large facilities have to solve sooner or later. And
the same problems must crop up whatever software is used (prman,
MentalRay etc). Following is a bunch of questions that come to mind,
some houdini specific, others more general.

Do you generate IFD's out on the farm or on artists workstations, and
if so how are resources synced across to where they are needed ?

Is it the case that you purchase expensive global file systems and
maintain a single shared file space ? If so, are they robust and
performant ?

Obviously their are significant performance issues to consider. Eg you
are working with Gb's of texture per frame which you don't want to
ship across to a render node every time a frame is rendered. There are
many possible solutions to this problem but I'd love to hear about
experiences in the trenches ...

At what point do you find that single NFS servers grind to halt ? Is
scalability and locality a solved problem or something you're always
having to tweak ?

Are you restricted to certain workstation architectures ?

Does your solution restrict you to a certain applications ?

Any feedback appreciated,

Drew Whitehouse
ANU Supercomputer Facility Vizlab

More information about the Sidefx-houdini-list mailing list