[Sidefx-houdini-list] Rendering Solutions... Any suggestions

Michael K. O'Brien mobrien at pixar.com
Thu Apr 28 15:58:45 EDT 2011


Hola~

>-- On Thu, Apr 28, 2011 at 08:25:18PM +0100, Lars van der Bijl wrote:
> but surely this is not the way task get submitted to hqueue?
> 
> I understand the method that houdini uses to create it dependency tree for
> local render. but how will it turn this into a tree of separate task to
> utilise the farm most efficiently?

That's what we do.

We take the output of the hscript render command and turn it into a parallel
job graph to alfred. We added two hdk nodes "In Parallel" and "In Serial" that
take up-stream rops and splits the render request into parallel or serial
requests.  Frame Dependency might be able to take the place of In Serial, but
don't think In Parallel. We use frame container, pre-post, and fetch quite a
bit as well.

This lets us run a simulation on a single farm machine, but the post
processing of the simulation results in parallel, for example.

The only non-houdini thing we do is to tag ROPs with parms that determine
machine, memory, mpi, etc... Those parms are eval-ed by the script that takes
the render output when generating an alfred script so alfred knows what sort
of profile to schedule on.

We also have a sop file-cache otl that can one-click send stuff to the farm.
This allows the user to iterate quickly, inline, and then create a true ROP
dependency graph using Fetch's.

It's been great. We offload nearly all non-trivial computation from the user's
workstation, which let's them multitask between shots on a single machine.

> 
> also something that bugs me in how subnets always get handeld as a "bucket"
> of task where everything that has no output connector gets rendered. this
> works perfectly in theory but it means that you cant split things out
> easily.

Yeah, we hit this for a long time. Subnets don't handle as bucket, but otl's
do. If you have a subnet, you can specify which inputs depend on up-stream
graphs.

We just stopped using ROP otl's and moved to toolbar buttons that generate a
subnet with a ROP graph in it. This allows us to specify which rops should
depend on up-stream connections and also let's users muck with the graph
easily (which we found they were doing quite a bit of in the otl case). So
far, that's been going well.

MO

> 
> stream one for local render. stream two for farm rendering. granted you can
> do it with a switch. set it to a invalid input. stopping the traversal.
> 
> it also means that it's harder to use houdini's traversal to create a
> dependency tree. or am I missing something?
> 
> Lars
> 
> 
> On Thu, Apr 28, 2011 at 7:42 PM, Mark Alexander <malexander at sidefx.com>wrote:
> 
> > You can generate a dependency tree from a ROP network using:
> >
> > render -p -I -f <start_fr> <end_fr>  <output_rop>
> >
> > -p will print the results instead of rendering
> >
> > -I will process all of frame 1 first, then frame 2, etc. instead of the
> > entire range of each ROP at once
> >
> > -f specifies the frame range.
> >
> > The output is in the format:
> >
> > <renderid> [ <dependencies> ]  <ROPname>  <frame>
> >
> > So, for a ROP named out, which depends on A and B (its inputs), rendering
> > frames 1 and 2 would look like:
> >
> > 1 [ ] A         1
> > 2 [ ] B         1
> > 3 [ 1 2 ] out   1
> > 4 [ ] A         2
> > 5 [ ] B         2
> > 6 [ 4 5 ] out   2
> >
> > So, render job 3 is rendering 'out' at frame 1, but is dependent on render
> > jobs 1 and 2.
> >
> > By default each frame depends on the corresponding frame number in the
> > inputs, but special ROPs such as Batch and Frame Dependency can alter the
> > frames that an output frame is dependent on. The Frame Container ROP can
> > restrict these frame dependency alterations, and the Pre Post ROP can
> > execute pre and post render ROPs, similar to using the pre and post render
> > scripts.
> >
> > Cheers,
> > M.
> >
> >
> > On 04/28/2011 02:31 PM, Peter Bowmar wrote:
> >
> >> Yes, dependencies for a render queue are an essential feature.
> >>
> >> On 28 April 2011 10:58, Lars van der Bijl<com48com at gmail.com>  wrote:
> >>
> >>> also looking at dependencies.
> >>>
> >>> is there a easy way of creating a dependency tree from a set of rops? one
> >>> of
> >>> the nice parts of the grid being that it has the concept of task but not
> >>> of
> >>> a "Job" (a collection of tasks). now initially this put me off and its
> >>> still
> >>> hard to visuals a collection of tasks for users but it means that create
> >>> dependencies between task is incredibly easy to do.
> >>>
> >>> here is a screen shot of a job we are doing now.
> >>>
> >>> this correlates 1 to 1 with a rop network. from pre-sim to final image.
> >>> (2
> >>> sets of images actually).
> >>>
> >>> is there any documentation on doing something similar with hqueue? I'd
> >>> love
> >>> to read up on it.
> >>>
> >>> Lars
> >>>
> >>>
> >>> On Thu, Apr 28, 2011 at 6:43 PM, Peter Bowmar<pbowmar at gmail.com>  wrote:
> >>>
> >>>  Hi Cristin,
> >>>>
> >>>> Curious, what's the largest farm that you know of using HQueue.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Peter B
> >>>>
> >>>> On 28 April 2011 09:59, Cristin Barghiel<cb at sidefx.com>  wrote:
> >>>>
> >>>>> Hi Phil,
> >>>>>
> >>>>> I encourage you to take a look at Houdini's HQueue. We are actively
> >>>>> using
> >>>>>
> >>>> it and supporting it, and it's already wired to work well with
> >>>> Houdini-native rendering and simulation. As it's python-based and
> >>>> architected to be open, it can handle rendering jobs - any kind of jobs,
> >>>> in
> >>>> fact - from packages other than Houdini as well.
> >>>>
> >>>>>
> >>>>> Cristin
> >>>>>
> >>>>> On 2011-04-28, at 4:51 AM, Phil Spicer wrote:
> >>>>>
> >>>>>  Hi,
> >>>>>>
> >>>>>> Our RenderFarm here at the NCCA has died, and we are looking for
> >>>>>>
> >>>>> alternative Render Solutions for the next academic year. Preferably it
> >>>> would
> >>>> be something we could lease on site, or we would lease time on the
> >>>> Amazon
> >>>> EC2 Compute Cloud.
> >>>>
> >>>>>
> >>>>>> I was wondering what people were using in Industry, and if anyone had
> >>>>>>
> >>>>> suggestions or information about what solution works best.
> >>>>
> >>>>>
> >>>>>> Kind regards,
> >>>>>>
> >>>>>> Phil.
> >>>>>>
> >>>>>> Philip Spicer
> >>>>>> Programme Coordinator - MA Digital Effects
> >>>>>> National Centre for Computer Animation
> >>>>>> Bournemouth University - UK
> >>>>>>
> >>>>>> http://www.youtube.com/user/NCCADigitalFX
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> This email is intended only for the person to whom it is addressed and
> >>>>>>
> >>>>> may contain confidential information. If you have received this email
> >>>> in
> >>>> error, please notify the sender and delete this email, which must not be
> >>>> copied, distributed or disclosed to any other person. otter
> >>>>
> >>>>> Any views or opinions presented are solely those of the author and do
> >>>>>>
> >>>>> not necessarily represent those of Bournemouth University or its
> >>>> subsidiary
> >>>> companies. Nor can any contract be formed on behalf of the University or
> >>>> its
> >>>> subsidiary companies via email.
> >>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Sidefx-houdini-list mailing list
> >>>>>> Sidefx-houdini-list at sidefx.com
> >>>>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>>>>>
> >>>>>
> >>>>> ---
> >>>>> C. Barghiel
> >>>>> Side Effects Software www.sidefx.com 416-504-9876
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Sidefx-houdini-list mailing list
> >>>>> Sidefx-houdini-list at sidefx.com
> >>>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> No, I am not on Facebook.
> >>>> _______________________________________________
> >>>> Sidefx-houdini-list mailing list
> >>>> Sidefx-houdini-list at sidefx.com
> >>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>>>
> >>>>
> >>> _______________________________________________
> >>> Sidefx-houdini-list mailing list
> >>> Sidefx-houdini-list at sidefx.com
> >>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>>
> >>>
> >>>
> >>
> >>
> >>
> > _______________________________________________
> > Sidefx-houdini-list mailing list
> > Sidefx-houdini-list at sidefx.com
> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >
> _______________________________________________
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list

-- 
Michael O'Brien                                                       FX Lead
Pixar                                                       mobrien at pixar.com



More information about the Sidefx-houdini-list mailing list