[Sidefx-houdini-list] render farms, pipelines and file systems

Moritz Moeller realritz at virtualritz.com
Mon Jan 12 17:12:10 EST 2009


jiversen wrote:
> Back at DD we tried something on Transformers: we had a Mantra wrapper
> script which examined the incoming IFD stream for files (specifically
> textures) and copied them to a local cache directory (if they didn't
> exist or were newer) and switch the paths inside the IFD to the local
> cache location. Megatron himself had about 2.5gb of textures.

This is not as clever as a cache lib working as a DSO since it copies 
files whether they are used or not.
For example, one limb of a robot might not be visibile to the camera but 
because it is referenced in a sequence of files, the textures etc. it 
will still be copied and possibly create even more traffic than if the 
frame was rendered using remote data.

My chachlib being exposed as a shadeop DSO avoid this problem but not 
entirely. I.e. if only one sample of a texture is queried, this causes 
copying the entite texture over. If this helps or make things work 
better depends on what you're rendering. Your milage may vary ...

> [...] I'm guessing due to the mip-level at that
> range and the hidden/backfaced texture tiles never being requested . So,
> it would seem that most efficient local caching would caching /only the
> tile and mip-level that is being requested/ - something that I believe
> only SESI could provide hooks for in their texture access functions.

See my above comment.
3Delight allows splitting textures into directory textures 
http://www.3delight.com/en/uploads/docs/3delight/3delight_55.html#SEC139.
Each mip-map level becomes an individual file. This upped granualrity 
can lead to huge further savings when using a cache.

But this is 3Delight specific and can only be replicated in other 
renderers if one uses a custom texture access shadeop one has source 
code access to.

> Also, Moritz:  I'm not sure if you're aware of the FS_ReaderHelper
> <http://odforce.net/hdk/tree/H9.1/classFS__ReaderHelper.html>  HDK
> class?  This would possibly make the best/most useful/most general way
> to intercept file accesses within Mantra (and Houdini!). Your Renderman
> DSO could possibly best be translated to this class instead of a VEXdso,
> perhaps,

This is a good point. Because to be really useful, such a cache should 
be used for *all* resources.
I.e. shaders, geometry archives, point clouds etc. This requires access 
to many hooks. In case of RMan renderers, a Rif filter to intercept all 
Ri.. calls that can request loading some sort file resource is needed.

3Delight's cache certainly does all this automatically (it also deals 
with delayed archives and exposes an Rx.. binding to its file caching 
for people writing DSOs that might load file resources).

That FS_ReadeHelper hook looks like the way to go for Houdini people. 
Users of PRMan & co will have no choice but to also write a Rif filter.

Another issue is write caching. On two shows I worked on in recent 
years, the number of AOVs and the fact that REYES-like renderers write 
finished buckets instead of finished frames, caused issues. Basically, 
you have something like 50-60 file handles for the AOVs per frame on the 
server, per blade. With a few hunderd blade on the farm dumping finished 
buckets (all small packets), the disks of the server become a much 
bigger bottlenck than the network. If they can't keep up, the write 
queue overflows and the resulting images are corrupt.

There is of course a hack: if you choose a bucket order that can't be 
replicated in the file format, most renderers cache files in RAM before 
writing them. If this hack really helps depends on how much RAM you have 
-- 50-60 float AOVs use quite a bit ...

On one of the two shows we used 3Delight and the guys added write 
caching for us.
Basically, displays get cached locally too and written back to the 
destination when the frame finishes. This causes a peak in network 
traffic. Apparently this is something much easier to deal with for the 
tech guys than the overflowing write queues on the server's disks. :)

Since my cache lib tries to replicate the 3Delight functionality, it 
supports write caching as well. But that part was never tested in 
production. :]


Beers,

Moritz



More information about the Sidefx-houdini-list mailing list