[Sidefx-houdini-list] render farms, pipelines and file systems
dbeeland at comcast.net
Thu Jan 15 00:50:30 EST 2009
Cachefs is read-only, at least in the R&H implementation. As Jason
mentioned, we write over the network via NFS mount, and don't have any
significant issues with it.
I've not read this entire thread, so my apologies if I've repeated
something already stated. :)
> Hey Moritz,
> At DD we'd do it your way too, writing the frame locally (in OpenEXR
> format) and then copying to the server when the frame was complete. It
> would add to the overall stability of the render because it wouldn't be
> so sensitive to dropped network mounts and other such SNAFUs.
> Here we write in R+H's proprietary deep format directly to the network
> and I haven't yet heard too much concern about network speed or
> stability issues affecting or being caused by this. As an artist it's
> really cool to be able to flipbook partially-completed renders. It lets
> you check your work during a render quite effectively. I suppose if
> there was a local-cache with a bit of check-pointing going on - like
> copy the frame over to the network every 15 mins - you might be able to
> get the best of both worlds.
> I don't know how /cachefs/ works with write caching... a question I need
> to ask, I suppose:)
> Moritz Moeller wrote:
>> Another issue is write caching. On two shows I worked on in recent
>> years, the number of AOVs and the fact that REYES-like renderers write
>> finished buckets instead of finished frames, caused issues. Basically,
>> you have something like 50-60 file handles for the AOVs per frame on the
>> server, per blade. With a few hunderd blade on the farm dumping finished
>> buckets (all small packets), the disks of the server become a much
>> bigger bottlenck than the network. If they can't keep up, the write
>> queue overflows and the resulting images are corrupt.
>> There is of course a hack: if you choose a bucket order that can't be
>> replicated in the file format, most renderers cache files in RAM before
>> writing them. If this hack really helps depends on how much RAM you have
>> -- 50-60 float AOVs use quite a bit ...
>> On one of the two shows we used 3Delight and the guys added write
>> caching for us.
>> Basically, displays get cached locally too and written back to the
>> destination when the frame finishes. This causes a peak in network
>> traffic. Apparently this is something much easier to deal with for the
>> tech guys than the overflowing write queues on the server's disks. :)
>> Since my cache lib tries to replicate the 3Delight functionality, it
>> supports write caching as well. But that part was never tested in
>> production. :]
>> Sidefx-houdini-list mailing list
>> Sidefx-houdini-list at sidefx.com
More information about the Sidefx-houdini-list