[Sidefx-houdini-list] render farms, pipelines and file systems

jiversen jiversen at rhythm.com
Mon Jan 12 17:52:47 EST 2009


Hey Moritz,

At DD we'd do it your way too, writing the frame locally (in OpenEXR 
format) and then copying to the server when the frame was complete. It 
would add to the overall stability of the render because it wouldn't be 
so sensitive to dropped network mounts and other such SNAFUs.

Here we write in R+H's proprietary deep format directly to the network 
and I haven't yet heard too much concern about network speed or 
stability issues affecting or being caused by this. As an artist it's 
really cool to be able to flipbook partially-completed renders. It lets 
you check your work during a render quite effectively.  I suppose if 
there was a local-cache with a bit of check-pointing going on - like 
copy the frame over to the network every 15 mins - you might be able to 
get the best of both worlds.

I don't know how /cachefs/ works with write caching... a question I need 
to ask, I suppose:)


Moritz Moeller wrote:
>
> Another issue is write caching. On two shows I worked on in recent 
> years, the number of AOVs and the fact that REYES-like renderers write 
> finished buckets instead of finished frames, caused issues. Basically, 
> you have something like 50-60 file handles for the AOVs per frame on the 
> server, per blade. With a few hunderd blade on the farm dumping finished 
> buckets (all small packets), the disks of the server become a much 
> bigger bottlenck than the network. If they can't keep up, the write 
> queue overflows and the resulting images are corrupt.
>
> There is of course a hack: if you choose a bucket order that can't be 
> replicated in the file format, most renderers cache files in RAM before 
> writing them. If this hack really helps depends on how much RAM you have 
> -- 50-60 float AOVs use quite a bit ...
>
> On one of the two shows we used 3Delight and the guys added write 
> caching for us.
> Basically, displays get cached locally too and written back to the 
> destination when the frame finishes. This causes a peak in network 
> traffic. Apparently this is something much easier to deal with for the 
> tech guys than the overflowing write queues on the server's disks. :)
>
> Since my cache lib tries to replicate the 3Delight functionality, it 
> supports write caching as well. But that part was never tested in 
> production. :]
>
>
> Beers,
>
> Moritz
> _______________________________________________
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>   

-- 
jason iversen
  fx supervisor
    r+h

 nork.




More information about the Sidefx-houdini-list mailing list