[Sidefx-houdini-list] render farms, pipelines and file systems

jiversen jiversen at rhythm.com
Mon Jan 12 14:39:59 EST 2009


Back at DD we tried something on Transformers: we had a Mantra wrapper 
script which examined the incoming IFD stream for files (specifically 
textures) and copied them to a local cache directory (if they didn't 
exist or were newer) and switch the paths inside the IFD to the local 
cache location. Megatron himself had about 2.5gb of textures.

We found this that every farm machine copying 2.5 gb of textures to the 
local resulted in a huge network load, hoping it would be a cost 
amortized by subsequent re-renders, which was partially true. 
Unfortunately we couldn't maintain such a large cache very long and we 
had to clear the caches too often- but this is something we anticipated 
running into.

What I didn't anticipate was the immense savings that the paged, tiled & 
mip-mapped nature of RAT accesses give you. Even having the entire 
Megatron in shot, allowing Mantra to access from the network directly 
was /much/ faster and easier on the network than the 2.5gb copy-to-local 
with local texture access: I'm guessing due to the mip-level at that 
range and the hidden/backfaced texture tiles never being requested . So, 
it would seem that most efficient local caching would caching /only the 
tile and mip-level that is being requested/ - something that I believe 
only SESI could provide hooks for in their texture access functions.

Also, Moritz:  I'm not sure if you're aware of the FS_ReaderHelper 
<http://odforce.net/hdk/tree/H9.1/classFS__ReaderHelper.html> HDK 
class?  This would possibly make the best/most useful/most general way 
to intercept file accesses within Mantra (and Houdini!). Your Renderman 
DSO could possibly best be translated to this class instead of a VEXdso, 
perhaps,


Mark Story wrote:
> Hi Moritz,
>
> Yea, toss the source my way, I'd love to see it.
>
> I've used Solaris' Cache FS in the past with good success, had no idea 
> it was ported to Linux land.  Thanks for the info Jason!
>
>
> Mark
>
>
>   
>> Paolo Berto wrote:
>>     
>>> in 3Delight RenderMan you can rely on the "network cache" for
>>> textures, maps and archives.
>>> All the details are here:
>>>
>>>      http://www.3delight.com/en/uploads/docs/3delight/3delight_55.html
>>>       
>> During my last job at a big facility in London, I wrote a cache lib that 
>> replicates the 3Delight network cache's behavior with some extras (e.g. 
>> # of retries). This also was exposed as a shadeop DSO in PRMan. The call 
>> is totally transparent to the user.
>>
>> e.g.:
>>
>>    color foo = texture(cacheFile(textureName));
>>
>> Since I wrote this in my spare time, the code is mine and I plan to make 
>> it OSS as soon as I find the time to look over the code again and clean 
>> it up a bit.
>>
>> The lib is thread-safe and the DSO is a new RSL plugin operating on 
>> grids. So any tests are just done once per grid and if one uses 
>> class-based shaders, it can be limited to once per shader instance.
>> I think it would be /very/ easy to port this to a VEX shadeop.
>>
>> Also comes with a scripting binding (tested with Python) via SWIG.
>>
>> If anyone's interested to use this b4 I find the time, I'm happy to 
>> throw a source archive your way. The lib should compile on any platform 
>> where boost is available. It uses boost::filesystem, so it should even 
>> work with weird windoze paths[tm].
>>
>>
>> Beers,
>>
>> Moritz
>> _______________________________________________
>> Sidefx-houdini-list mailing list
>> Sidefx-houdini-list at sidefx.com
>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>     
>
> _______________________________________________
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>   

-- 
jason iversen
  fx supervisor
    r+h

 nork.




More information about the Sidefx-houdini-list mailing list