[Sidefx-houdini-list] render farms, pipelines and file systems

Jeff Beeland dbeeland at comcast.net
Thu Jan 15 01:08:49 EST 2009


I should be a little more specific, actually.  We don't have any 
significant issues with it any more.  A few years back we had fairly 
serious problems with hung mounts creeping up during heavier usage, as 
well as the queue simply overwhelming the disks.  Systems tossed out 
automount and rolled their own version that took care of the former (via 
magic I'm not versed in), and the latter has been largely eliminated by 
overhauling the studio filesystem to include multiple Isilon clusters, 
which shoulder the load quite well.  The cachefs system Jason mentioned 
is icing on the cake, and as it's rolled out will further lessen the 
burden on the filesystem.

--Jeff



Jeff Beeland wrote:
> Cachefs is read-only, at least in the R&H implementation.  As Jason 
> mentioned, we write over the network via NFS mount, and don't have any 
> significant issues with it.
>
> I've not read this entire thread, so my apologies if I've repeated 
> something already stated. :)
>
> --Jeff
>
>
>
> jiversen wrote:
>   
>> Hey Moritz,
>>
>> At DD we'd do it your way too, writing the frame locally (in OpenEXR 
>> format) and then copying to the server when the frame was complete. It 
>> would add to the overall stability of the render because it wouldn't be 
>> so sensitive to dropped network mounts and other such SNAFUs.
>>
>> Here we write in R+H's proprietary deep format directly to the network 
>> and I haven't yet heard too much concern about network speed or 
>> stability issues affecting or being caused by this. As an artist it's 
>> really cool to be able to flipbook partially-completed renders. It lets 
>> you check your work during a render quite effectively.  I suppose if 
>> there was a local-cache with a bit of check-pointing going on - like 
>> copy the frame over to the network every 15 mins - you might be able to 
>> get the best of both worlds.
>>
>> I don't know how /cachefs/ works with write caching... a question I need 
>> to ask, I suppose:)
>>
>>
>> Moritz Moeller wrote:
>>   
>>     
>>> Another issue is write caching. On two shows I worked on in recent 
>>> years, the number of AOVs and the fact that REYES-like renderers write 
>>> finished buckets instead of finished frames, caused issues. Basically, 
>>> you have something like 50-60 file handles for the AOVs per frame on the 
>>> server, per blade. With a few hunderd blade on the farm dumping finished 
>>> buckets (all small packets), the disks of the server become a much 
>>> bigger bottlenck than the network. If they can't keep up, the write 
>>> queue overflows and the resulting images are corrupt.
>>>
>>> There is of course a hack: if you choose a bucket order that can't be 
>>> replicated in the file format, most renderers cache files in RAM before 
>>> writing them. If this hack really helps depends on how much RAM you have 
>>> -- 50-60 float AOVs use quite a bit ...
>>>
>>> On one of the two shows we used 3Delight and the guys added write 
>>> caching for us.
>>> Basically, displays get cached locally too and written back to the 
>>> destination when the frame finishes. This causes a peak in network 
>>> traffic. Apparently this is something much easier to deal with for the 
>>> tech guys than the overflowing write queues on the server's disks. :)
>>>
>>> Since my cache lib tries to replicate the 3Delight functionality, it 
>>> supports write caching as well. But that part was never tested in 
>>> production. :]
>>>
>>>
>>> Beers,
>>>
>>> Moritz
>>> _______________________________________________
>>> Sidefx-houdini-list mailing list
>>> Sidefx-houdini-list at sidefx.com
>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>>   
>>>     
>>>       
>>   
>>     
> _______________________________________________
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>
>   



More information about the Sidefx-houdini-list mailing list