[Sidefx-houdini-list] Rendering Solutions... Any suggestions

Michael K. O'Brien mobrien at pixar.com
Thu Apr 28 18:34:58 EDT 2011


>-- On Thu, Apr 28, 2011 at 10:33:08PM +0100, Lars van der Bijl wrote:
> On Thu, Apr 28, 2011 at 8:58 PM, Michael K. O'Brien <mobrien at pixar.com>wrote:
> > Hola~
> >
> > >-- On Thu, Apr 28, 2011 at 08:25:18PM +0100, Lars van der Bijl wrote:
> > > but surely this is not the way task get submitted to hqueue?
> > >
> > > I understand the method that houdini uses to create it dependency tree
> > for
> > > local render. but how will it turn this into a tree of separate task to
> > > utilise the farm most efficiently?
> >
> > That's what we do.
> >
> > We take the output of the hscript render command and turn it into a
> > parallel
> > job graph to alfred. We added two hdk nodes "In Parallel" and "In Serial"
> > that
> > take up-stream rops and splits the render request into parallel or serial
> > requests.  Frame Dependency might be able to take the place of In Serial,
> > but
> > don't think In Parallel. We use frame container, pre-post, and fetch quite
> > a
> > bit as well.
> >
> > This lets us run a simulation on a single farm machine, but the post
> > processing of the simulation results in parallel, for example.
> >
> > The only non-houdini thing we do is to tag ROPs with parms that determine
> > machine, memory, mpi, etc... Those parms are eval-ed by the script that
> > takes
> > the render output when generating an alfred script so alfred knows what
> > sort
> > of profile to schedule on.
> >
> > We also have a sop file-cache otl that can one-click send stuff to the
> > farm.
> > This allows the user to iterate quickly, inline, and then create a true ROP
> > dependency graph using Fetch's.
> >
> > It's been great. We offload nearly all non-trivial computation from the
> > user's
> > workstation, which let's them multitask between shots on a single machine.
> >
> >
> that is interesting. we have simular functionality with our current
> implementation but we ended up writing own traversal.
> giving us the same control with a "cache" rop/sop that can be submitted on
> chained.

Yeah, I know a couple of studios that ended up writing traversal code. I
really didn't want to own that, and like the switch, fetch, pre/post, and
frame container. I really want to let SideFX do cool things there and have it
just work, and not feel like I need to update our traversal code everytime a
new ROP comes out.

We do a large number of things this way including simulation, geometry
caching, asset creation, building, rendering using our interal tools,
installing, etc... The typical graph for an interactive session is usually
fairly contained, but for an fx rig can get to be huge.

For example, the asset creation has a shader export, geometry export, model
export (like a template), and a build step. The user often wants to iterate on
the shader export + build (happens once per render request), or maybe the
geometry export (geom is per frame, build is once per request). As an otl,
users ended up doing things like tweaking dependencies and tagging just the
geometry rop with high-memory and proc requirements for the farm.

> we have our setting on a otl rop. i've implemented a simulare system as you
> describe before and found it difficult to promise backward compatibility. of
> course as i was developing it these parameters where in flux but ones the
> setting settle down it worked fine. having a otl at leased means i don't
> have to bug the artist to update there parms on there rops.
> i'll have to have a other look at the hscript render.
> >
> > > also something that bugs me in how subnets always get handeld as a
> > "bucket"
> > > of task where everything that has no output connector gets rendered. this
> > > works perfectly in theory but it means that you cant split things out
> > > easily.
> >
> > Yeah, we hit this for a long time. Subnets don't handle as bucket, but
> > otl's
> > do. If you have a subnet, you can specify which inputs depend on up-stream
> > graphs.
> >
> > We just stopped using ROP otl's and moved to toolbar buttons that generate
> > a
> > subnet with a ROP graph in it. This allows us to specify which rops should
> > depend on up-stream connections and also let's users muck with the graph
> > easily (which we found they were doing quite a bit of in the otl case). So
> > far, that's been going well.
> >
> > MO
> >
> >
> my grievance was with downstream how it's harder to specify which output
> should be chosen from within a subnet or a otl. both otl and subnet have
> input pegs (if you set the otls to have inputs of course).

We have our rops setup to not require input (0 min) which means you don't get
input pegs.

We author the ROP graphs bottom up, which is the reverse from most of the
other graphs in Houdini, so we always have one output point for downstream
nodes to correctly wire in.

> however the move away from otls is intriguing though. do you have your own
> tools to add stuff to the toolbar or are you using the default xml file? i

Just the default xml files. We have a source/inst split. So, you point Houdini
at your source tree, use Houdini to edit, then build it out to inst. You can
also just edit using vim as well.

It's a bit clunky the first time, but doesn't really slow me down compared to
writing the Python code to build the network.


> have found this difficult to manage using the default xmls and ended up
> writing tools
> to create the toolbars dynamically using python. mainly because of our
> software install system which made it harder to define paths for icon's
> before hand.
    (snip) 8<----

Michael O'Brien                                                       FX Lead
Pixar                                                       mobrien at pixar.com

More information about the Sidefx-houdini-list mailing list