[Sidefx-houdini-list] render farms, pipelines and file systems
Andrew D Lyons
tstexture at gmail.com
Mon Jan 12 15:52:30 EST 2009
Interesting stuff. I'm guessing you could solve the reyes bucket
related bottleneck by writing to the local temp drive during render,
and then doing a copy at the end when the frame is complete and
2009/1/12 Moritz Moeller <realritz at virtualritz.com>:
> jiversen wrote:
>> Back at DD we tried something on Transformers: we had a Mantra wrapper
>> script which examined the incoming IFD stream for files (specifically
>> textures) and copied them to a local cache directory (if they didn't
>> exist or were newer) and switch the paths inside the IFD to the local
>> cache location. Megatron himself had about 2.5gb of textures.
> This is not as clever as a cache lib working as a DSO since it copies
> files whether they are used or not.
> For example, one limb of a robot might not be visibile to the camera but
> because it is referenced in a sequence of files, the textures etc. it
> will still be copied and possibly create even more traffic than if the
> frame was rendered using remote data.
> My chachlib being exposed as a shadeop DSO avoid this problem but not
> entirely. I.e. if only one sample of a texture is queried, this causes
> copying the entite texture over. If this helps or make things work
> better depends on what you're rendering. Your milage may vary ...
>> [...] I'm guessing due to the mip-level at that
>> range and the hidden/backfaced texture tiles never being requested . So,
>> it would seem that most efficient local caching would caching /only the
>> tile and mip-level that is being requested/ - something that I believe
>> only SESI could provide hooks for in their texture access functions.
> See my above comment.
> 3Delight allows splitting textures into directory textures
> Each mip-map level becomes an individual file. This upped granualrity
> can lead to huge further savings when using a cache.
> But this is 3Delight specific and can only be replicated in other
> renderers if one uses a custom texture access shadeop one has source
> code access to.
>> Also, Moritz: I'm not sure if you're aware of the FS_ReaderHelper
>> <http://odforce.net/hdk/tree/H9.1/classFS__ReaderHelper.html> HDK
>> class? This would possibly make the best/most useful/most general way
>> to intercept file accesses within Mantra (and Houdini!). Your Renderman
>> DSO could possibly best be translated to this class instead of a VEXdso,
> This is a good point. Because to be really useful, such a cache should
> be used for *all* resources.
> I.e. shaders, geometry archives, point clouds etc. This requires access
> to many hooks. In case of RMan renderers, a Rif filter to intercept all
> Ri.. calls that can request loading some sort file resource is needed.
> 3Delight's cache certainly does all this automatically (it also deals
> with delayed archives and exposes an Rx.. binding to its file caching
> for people writing DSOs that might load file resources).
> That FS_ReadeHelper hook looks like the way to go for Houdini people.
> Users of PRMan & co will have no choice but to also write a Rif filter.
> Another issue is write caching. On two shows I worked on in recent
> years, the number of AOVs and the fact that REYES-like renderers write
> finished buckets instead of finished frames, caused issues. Basically,
> you have something like 50-60 file handles for the AOVs per frame on the
> server, per blade. With a few hunderd blade on the farm dumping finished
> buckets (all small packets), the disks of the server become a much
> bigger bottlenck than the network. If they can't keep up, the write
> queue overflows and the resulting images are corrupt.
> There is of course a hack: if you choose a bucket order that can't be
> replicated in the file format, most renderers cache files in RAM before
> writing them. If this hack really helps depends on how much RAM you have
> -- 50-60 float AOVs use quite a bit ...
> On one of the two shows we used 3Delight and the guys added write
> caching for us.
> Basically, displays get cached locally too and written back to the
> destination when the frame finishes. This causes a peak in network
> traffic. Apparently this is something much easier to deal with for the
> tech guys than the overflowing write queues on the server's disks. :)
> Since my cache lib tries to replicate the 3Delight functionality, it
> supports write caching as well. But that part was never tested in
> production. :]
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
Andrew D Lyons | Digital Artist | http://www.tstex.com
More information about the Sidefx-houdini-list