[Sidefx-houdini-list] Rendering Solutions... Any suggestions

Lars van der Bijl com48com at gmail.com
Thu Apr 28 17:33:08 EDT 2011


On Thu, Apr 28, 2011 at 8:58 PM, Michael K. O'Brien <mobrien at pixar.com>wrote:

> Hola~
>
> >-- On Thu, Apr 28, 2011 at 08:25:18PM +0100, Lars van der Bijl wrote:
> > but surely this is not the way task get submitted to hqueue?
> >
> > I understand the method that houdini uses to create it dependency tree
> for
> > local render. but how will it turn this into a tree of separate task to
> > utilise the farm most efficiently?
>
> That's what we do.
>
> We take the output of the hscript render command and turn it into a
> parallel
> job graph to alfred. We added two hdk nodes "In Parallel" and "In Serial"
> that
> take up-stream rops and splits the render request into parallel or serial
> requests.  Frame Dependency might be able to take the place of In Serial,
> but
> don't think In Parallel. We use frame container, pre-post, and fetch quite
> a
> bit as well.
>
> This lets us run a simulation on a single farm machine, but the post
> processing of the simulation results in parallel, for example.
>
> The only non-houdini thing we do is to tag ROPs with parms that determine
> machine, memory, mpi, etc... Those parms are eval-ed by the script that
> takes
> the render output when generating an alfred script so alfred knows what
> sort
> of profile to schedule on.
>
> We also have a sop file-cache otl that can one-click send stuff to the
> farm.
> This allows the user to iterate quickly, inline, and then create a true ROP
> dependency graph using Fetch's.
>
> It's been great. We offload nearly all non-trivial computation from the
> user's
> workstation, which let's them multitask between shots on a single machine.
>
>
that is interesting. we have simular functionality with our current
implementation but we ended up writing own traversal.
giving us the same control with a "cache" rop/sop that can be submitted on
chained.

we have our setting on a otl rop. i've implemented a simulare system as you
describe before and found it difficult to promise backward compatibility. of
course as i was developing it these parameters where in flux but ones the
setting settle down it worked fine. having a otl at leased means i don't
have to bug the artist to update there parms on there rops.

i'll have to have a other look at the hscript render.


>
> > also something that bugs me in how subnets always get handeld as a
> "bucket"
> > of task where everything that has no output connector gets rendered. this
> > works perfectly in theory but it means that you cant split things out
> > easily.
>
> Yeah, we hit this for a long time. Subnets don't handle as bucket, but
> otl's
> do. If you have a subnet, you can specify which inputs depend on up-stream
> graphs.
>
> We just stopped using ROP otl's and moved to toolbar buttons that generate
> a
> subnet with a ROP graph in it. This allows us to specify which rops should
> depend on up-stream connections and also let's users muck with the graph
> easily (which we found they were doing quite a bit of in the otl case). So
> far, that's been going well.
>
> MO
>
>
my grievance was with downstream how it's harder to specify which output
should be chosen from within a subnet or a otl. both otl and subnet have
input pegs (if you set the otls to have inputs of course).
however the move away from otls is intriguing though. do you have your own
tools to add stuff to the toolbar or are you using the default xml file? i
have found this difficult to manage using the default xmls and ended up
writing tools
to create the toolbars dynamically using python. mainly because of our
software install system which made it harder to define paths for icon's
before hand.



>
> > stream one for local render. stream two for farm rendering. granted you
> can
> > do it with a switch. set it to a invalid input. stopping the traversal.
> >
> > it also means that it's harder to use houdini's traversal to create a
> > dependency tree. or am I missing something?
> >
> > Lars
> >
> >
> > On Thu, Apr 28, 2011 at 7:42 PM, Mark Alexander <malexander at sidefx.com
> >wrote:
> >
> > > You can generate a dependency tree from a ROP network using:
> > >
> > > render -p -I -f <start_fr> <end_fr>  <output_rop>
> > >
> > > -p will print the results instead of rendering
> > >
> > > -I will process all of frame 1 first, then frame 2, etc. instead of the
> > > entire range of each ROP at once
> > >
> > > -f specifies the frame range.
> > >
> > > The output is in the format:
> > >
> > > <renderid> [ <dependencies> ]  <ROPname>  <frame>
> > >
> > > So, for a ROP named out, which depends on A and B (its inputs),
> rendering
> > > frames 1 and 2 would look like:
> > >
> > > 1 [ ] A         1
> > > 2 [ ] B         1
> > > 3 [ 1 2 ] out   1
> > > 4 [ ] A         2
> > > 5 [ ] B         2
> > > 6 [ 4 5 ] out   2
> > >
> > > So, render job 3 is rendering 'out' at frame 1, but is dependent on
> render
> > > jobs 1 and 2.
> > >
> > > By default each frame depends on the corresponding frame number in the
> > > inputs, but special ROPs such as Batch and Frame Dependency can alter
> the
> > > frames that an output frame is dependent on. The Frame Container ROP
> can
> > > restrict these frame dependency alterations, and the Pre Post ROP can
> > > execute pre and post render ROPs, similar to using the pre and post
> render
> > > scripts.
> > >
> > > Cheers,
> > > M.
> > >
> > >
> > > On 04/28/2011 02:31 PM, Peter Bowmar wrote:
> > >
> > >> Yes, dependencies for a render queue are an essential feature.
> > >>
> > >> On 28 April 2011 10:58, Lars van der Bijl<com48com at gmail.com>  wrote:
> > >>
> > >>> also looking at dependencies.
> > >>>
> > >>> is there a easy way of creating a dependency tree from a set of rops?
> one
> > >>> of
> > >>> the nice parts of the grid being that it has the concept of task but
> not
> > >>> of
> > >>> a "Job" (a collection of tasks). now initially this put me off and
> its
> > >>> still
> > >>> hard to visuals a collection of tasks for users but it means that
> create
> > >>> dependencies between task is incredibly easy to do.
> > >>>
> > >>> here is a screen shot of a job we are doing now.
> > >>>
> > >>> this correlates 1 to 1 with a rop network. from pre-sim to final
> image.
> > >>> (2
> > >>> sets of images actually).
> > >>>
> > >>> is there any documentation on doing something similar with hqueue?
> I'd
> > >>> love
> > >>> to read up on it.
> > >>>
> > >>> Lars
> > >>>
> > >>>
> > >>> On Thu, Apr 28, 2011 at 6:43 PM, Peter Bowmar<pbowmar at gmail.com>
>  wrote:
> > >>>
> > >>>  Hi Cristin,
> > >>>>
> > >>>> Curious, what's the largest farm that you know of using HQueue.
> > >>>>
> > >>>> Cheers,
> > >>>>
> > >>>> Peter B
> > >>>>
> > >>>> On 28 April 2011 09:59, Cristin Barghiel<cb at sidefx.com>  wrote:
> > >>>>
> > >>>>> Hi Phil,
> > >>>>>
> > >>>>> I encourage you to take a look at Houdini's HQueue. We are actively
> > >>>>> using
> > >>>>>
> > >>>> it and supporting it, and it's already wired to work well with
> > >>>> Houdini-native rendering and simulation. As it's python-based and
> > >>>> architected to be open, it can handle rendering jobs - any kind of
> jobs,
> > >>>> in
> > >>>> fact - from packages other than Houdini as well.
> > >>>>
> > >>>>>
> > >>>>> Cristin
> > >>>>>
> > >>>>> On 2011-04-28, at 4:51 AM, Phil Spicer wrote:
> > >>>>>
> > >>>>>  Hi,
> > >>>>>>
> > >>>>>> Our RenderFarm here at the NCCA has died, and we are looking for
> > >>>>>>
> > >>>>> alternative Render Solutions for the next academic year. Preferably
> it
> > >>>> would
> > >>>> be something we could lease on site, or we would lease time on the
> > >>>> Amazon
> > >>>> EC2 Compute Cloud.
> > >>>>
> > >>>>>
> > >>>>>> I was wondering what people were using in Industry, and if anyone
> had
> > >>>>>>
> > >>>>> suggestions or information about what solution works best.
> > >>>>
> > >>>>>
> > >>>>>> Kind regards,
> > >>>>>>
> > >>>>>> Phil.
> > >>>>>>
> > >>>>>> Philip Spicer
> > >>>>>> Programme Coordinator - MA Digital Effects
> > >>>>>> National Centre for Computer Animation
> > >>>>>> Bournemouth University - UK
> > >>>>>>
> > >>>>>> http://www.youtube.com/user/NCCADigitalFX
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> This email is intended only for the person to whom it is addressed
> and
> > >>>>>>
> > >>>>> may contain confidential information. If you have received this
> email
> > >>>> in
> > >>>> error, please notify the sender and delete this email, which must
> not be
> > >>>> copied, distributed or disclosed to any other person. otter
> > >>>>
> > >>>>> Any views or opinions presented are solely those of the author and
> do
> > >>>>>>
> > >>>>> not necessarily represent those of Bournemouth University or its
> > >>>> subsidiary
> > >>>> companies. Nor can any contract be formed on behalf of the
> University or
> > >>>> its
> > >>>> subsidiary companies via email.
> > >>>>
> > >>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> _______________________________________________
> > >>>>>> Sidefx-houdini-list mailing list
> > >>>>>> Sidefx-houdini-list at sidefx.com
> > >>>>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> > >>>>>>
> > >>>>>
> > >>>>> ---
> > >>>>> C. Barghiel
> > >>>>> Side Effects Software www.sidefx.com 416-504-9876
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> Sidefx-houdini-list mailing list
> > >>>>> Sidefx-houdini-list at sidefx.com
> > >>>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> No, I am not on Facebook.
> > >>>> _______________________________________________
> > >>>> Sidefx-houdini-list mailing list
> > >>>> Sidefx-houdini-list at sidefx.com
> > >>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> > >>>>
> > >>>>
> > >>> _______________________________________________
> > >>> Sidefx-houdini-list mailing list
> > >>> Sidefx-houdini-list at sidefx.com
> > >>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > > _______________________________________________
> > > Sidefx-houdini-list mailing list
> > > Sidefx-houdini-list at sidefx.com
> > > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> > >
> > _______________________________________________
> > Sidefx-houdini-list mailing list
> > Sidefx-houdini-list at sidefx.com
> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>
> --
> Michael O'Brien                                                       FX
> Lead
> Pixar
> mobrien at pixar.com
> _______________________________________________
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>



More information about the Sidefx-houdini-list mailing list