[Sidefx-houdini-list] render farms, pipelines and file systems
jiversen at rhythm.com
Mon Jan 12 14:39:59 EST 2009
Back at DD we tried something on Transformers: we had a Mantra wrapper
script which examined the incoming IFD stream for files (specifically
textures) and copied them to a local cache directory (if they didn't
exist or were newer) and switch the paths inside the IFD to the local
cache location. Megatron himself had about 2.5gb of textures.
We found this that every farm machine copying 2.5 gb of textures to the
local resulted in a huge network load, hoping it would be a cost
amortized by subsequent re-renders, which was partially true.
Unfortunately we couldn't maintain such a large cache very long and we
had to clear the caches too often- but this is something we anticipated
What I didn't anticipate was the immense savings that the paged, tiled &
mip-mapped nature of RAT accesses give you. Even having the entire
Megatron in shot, allowing Mantra to access from the network directly
was /much/ faster and easier on the network than the 2.5gb copy-to-local
with local texture access: I'm guessing due to the mip-level at that
range and the hidden/backfaced texture tiles never being requested . So,
it would seem that most efficient local caching would caching /only the
tile and mip-level that is being requested/ - something that I believe
only SESI could provide hooks for in their texture access functions.
Also, Moritz: I'm not sure if you're aware of the FS_ReaderHelper
class? This would possibly make the best/most useful/most general way
to intercept file accesses within Mantra (and Houdini!). Your Renderman
DSO could possibly best be translated to this class instead of a VEXdso,
Mark Story wrote:
> Hi Moritz,
> Yea, toss the source my way, I'd love to see it.
> I've used Solaris' Cache FS in the past with good success, had no idea
> it was ported to Linux land. Thanks for the info Jason!
>> Paolo Berto wrote:
>>> in 3Delight RenderMan you can rely on the "network cache" for
>>> textures, maps and archives.
>>> All the details are here:
>> During my last job at a big facility in London, I wrote a cache lib that
>> replicates the 3Delight network cache's behavior with some extras (e.g.
>> # of retries). This also was exposed as a shadeop DSO in PRMan. The call
>> is totally transparent to the user.
>> color foo = texture(cacheFile(textureName));
>> Since I wrote this in my spare time, the code is mine and I plan to make
>> it OSS as soon as I find the time to look over the code again and clean
>> it up a bit.
>> The lib is thread-safe and the DSO is a new RSL plugin operating on
>> grids. So any tests are just done once per grid and if one uses
>> class-based shaders, it can be limited to once per shader instance.
>> I think it would be /very/ easy to port this to a VEX shadeop.
>> Also comes with a scripting binding (tested with Python) via SWIG.
>> If anyone's interested to use this b4 I find the time, I'm happy to
>> throw a source archive your way. The lib should compile on any platform
>> where boost is available. It uses boost::filesystem, so it should even
>> work with weird windoze paths[tm].
>> Sidefx-houdini-list mailing list
>> Sidefx-houdini-list at sidefx.com
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
More information about the Sidefx-houdini-list