[Sidefx-houdini-list] HDK: op:/ magic

Olex P hoknamahn at gmail.com
Fri Jan 21 20:42:04 EST 2011


Yes, test was heavy enough (1024x1024). So in theory it had to be running on
8 threads. But top gave me 15 to 90 percent of CPU used so I'm not sure was
it really multithreaded. Pprobably it makes sense to do one more test that
gives you the id of a thread it's running on.
GDP object was allocated in Init(). Same with loading of geometry.
I think you can also do first loading in Evaluate() in order to be able to
use the file name passed to VEX op. Shouldn't be a problem. I'll check that
tomorrow.

On Sat, Jan 22, 2011 at 12:17 AM, Szymon Kapeniak <szymon.kapeniak at gmail.com
> wrote:

> That would be fantastic! But is your geometry enough heavy to start
> multi-threading? Grid 100x100 should be fine, but anything smaller
> doesn't use threads even if you set it on a vex node.
>
> Secondly, do you allocate new GU_Detail and load bgeo inside init()? I
> would love to do that, but I don't know how to pass user arguments
> there. It doesn't seem to take any, so all I have must take place
> inside main function. I may be wrong also in this point though...
>
> I'll try your example tomorrow.
>
> Thank you Olex!
>
> 2011/1/22 Olex P <hoknamahn at gmail.com>:
> > Hi Szymon,
> >
> > I've made a simple VEX dso which creates a static pointer to GU_Detail
> >
> > static GU_Detail* geo = NULL;
> >
> > allocates new object on Init()
> >
> > if(!geo)
> > {
> >  geo = new GU_Detail();
> >  geo->load("/tmp/something.bgeo", 0);
> > }
> >
> > takes an attribute value on Evaluate and returns it
> >
> > GEO_Point* pt = geo->points()[ptnum];
> > GB_AttributeRef somethingIndex = geo->findPointAttrib("something",
> > GB_ATTRIB_FLOAT);
> > const float* something = pt->castAttribData<float>(somethingIndex);
> > *result = *something;
> >
> > VEX op itself looks like:
> >
> > sop
> > vextest(export float newsomething = 0)
> > {
> >    newsomething = myfun((float)ptnum);
> > }
> >
> > So we create a geometry, apply an attribute on it, save somewhere. Then
> > apply this VEX op to clean (without an attribute geo) and spreadsheet
> show
> > exactly the same values as where written into bgeo.
> > It works fine.
> > On Fri, Jan 21, 2011 at 6:30 PM, Szymon Kapeniak
> > <szymon.kapeniak at gmail.com>wrote:
> >
> >> The problem is I don't have a chance to access any attribute, read,
> >> write, what so ever. I declare gdp as static pointer and create new
> >> GU_Detail inside init(), which meant to run once before threads start,
> >> then I point threads to that gdp as advised in docs. It's enough to
> >> crash Houdini in sop context.
> >>
> >> 2011/1/21 Olex P <hoknamahn at gmail.com>:
> >> > Hi Szymon,
> >> >
> >> > VEX itself has nothing to do with multithreading. But VEX engine
> >> (standard
> >> > contexts or your own CVEX one) does. So in the case you mentioned (VEX
> >> SOP
> >> > that is reading a point cloud and assigning a filtered value to a
> point)
> >> it
> >> > works as (simplified):
> >> > 1. VEX engine (SOP context) creates a flat array and fills it in with
> the
> >> > attributes from geometry
> >> > 2. VEX engine spawns several threads and runs one instance of VEX
> >> procedure
> >> > in each thread
> >> > 3. Each instance of VEX procedure (single threaded) filters a value
> from
> >> > point cloud and writes into it's chunk of flat array
> >> > 4. VEX engine writes all the values back from flat array into a GDP
> using
> >> > one thread
> >> > Honestly I haven't tried multithreading with GDP but I see no problems
> >> with
> >> > it as long as you are not modifying the data. And normally VEX
> function
> >> > itself doesn't modify a GDP (but VEX engine does).
> >> > Well that's how I understand it and I can be wrong :)
> >> > There are some bits about thread safety in HDK
> >> >
> >>
> http://www.sidefx.com/docs/hdk11.0/hdk_changes_11_0.html#HDKC11_0_ThreadSafety
> >> > I probably have to do some tests too :)
> >> >
> >> > On Fri, Jan 21, 2011 at 5:13 PM, Szymon Kapeniak
> >> > <szymon.kapeniak at gmail.com>wrote:
> >> >
> >> >> Hey Olex,
> >> >> I'm writing custom vex, and I'm sure vex has some standard methods to
> >> >> deal with this. I'm only interested in read access from several
> >> >> threads which I don't have any control on, as they are spawned by VEX
> >> >> SOP.
> >> >>
> >> >> I started to play with it by checking minimal scenarios and it seems
> >> >> it's enough to declare one static GU_Detail instance and assign
> >> >> pointers to it from a couple of threads to crash Houdini even without
> >> >> loading any geometry.
> >> >> Also by checking threads ids I realized, that mantra can run the same
> >> >> code multi-threaded because vex in surface context call init()
> >> >> (standard procedure) per thread, unlike in sop context, where it
> seems
> >> >> init() is called only once like in fact it's described in docs.
> >> >>
> >> >> Surly there are thread-save vex functions (reading gdp) which operate
> >> >> in sop context, like pcopen().  I'm guessing the trick is not to use
> >> >> pure gdp, but some  thread-safe memory schema. Lack of documentation
> >> >> doesn't play well with my ignorance...
> >> >>
> >> >> skk.
> >> >>
> >> >> btw  I'm referring to this talk:
> >> >>
> >> >>
> >>
> http://forums.odforce.net/index.php?/topic/12434-how-to-replicate-this-ice-network-in-vops/page__st__12
> >> >>
> >> >> 2011/1/21 Olex P <hoknamahn at gmail.com>:
> >> >> > Hey Szymon,
> >> >> >
> >> >> > You can read in multiple threads but multithread writing as far as
> I
> >> know
> >> >> is
> >> >> > limited to volume primitives (each thread can write into it's own
> >> tile)
> >> >> as
> >> >> > far as I know. You could create own GU/GEO_Detail object for each
> >> thread,
> >> >> > write into it and then merge all the pieces of puzzle into one
> object
> >> >> (merge
> >> >> > will single thread).
> >> >> > But from your words I'm not sure what kind of parallelism you are
> >> looking
> >> >> > for...
> >> >> > Let's say sampling from volume in VEX. 1. You are running the same
> >> single
> >> >> > thread code on different chunks of data using multiple threads. 2.
> But
> >> >> this
> >> >> > also could be done in a different way: vex function internally
> spawns
> >> >> > several threads which modify the data somehow. You probably talking
> >> about
> >> >> 2
> >> >> > (as you mentioned multithread access to GU_Detail). I'm not sure
> but
> >> >> maybe
> >> >> > it makes sense to use 1 (i.e. CVEX). Sure it will work only for
> data
> >> >> > parallel algorithms.
> >> >> >
> >> >> > On Thu, Jan 20, 2011 at 10:20 PM, Szymon Kapeniak <
> >> >> szymon.kapeniak at gmail.com
> >> >> >> wrote:
> >> >> >
> >> >> >> ...sorry for spamming, but I've just realized that the same dso
> works
> >> >> >> in mantra, but not in (threading) sop context. Kind of strange.
> >> >> >>
> >> >> >> 2011/1/20 Szymon Kapeniak <szymon.kapeniak at gmail.com>:
> >> >> >> > Thanks for a response, I didn't know I have to deal with it
> >> manually.
> >> >> >> >
> >> >> >> > Second problem is threading in vex sop. After playing with my
> >> function
> >> >> >> > a little, I'm getting a feeling my approach was too naive, so to
> >> >> >> > speak.
> >> >> >> >
> >> >> >> > Is multi-threaded access to a static GU_Detail * possible at
> all,
> >> or
> >> >> >> > do I have to use some caches like UT_ThreadSafeCache or anything
> >> >> >> > similar? What is a general way of doing this in HDK?
> >> >> >> >
> >> >> >> > thanks,
> >> >> >> > skk.
> >> >> >> >
> >> >> >> >
> >> >> >> > 2011/1/19 Olex P <hoknamahn at gmail.com>:
> >> >> >> >> It kind of does writing an op-referenced texture into an IFD :)
> >> >> >> >>
> >> >> >> >> On Wed, Jan 19, 2011 at 1:48 PM, Mark Elendt <mark at sidefx.com>
> >> >> wrote:
> >> >> >> >>
> >> >> >> >>> For mantra, even Houdini's VEX doesn't support the op: syntax
> :)
> >> >> >> >>>
> >> >> >> >>> For VEX functions in OPs, you should be able to access the SOP
> by
> >> >> >> >>> grabbing the OP director and finding the SOP.
> >> >> >> >>>
> >> >> >> >>> On Wednesday Jan 19 at 13:39, Szymon Kapeniak wrote:
> >> >> >> >>> > thanks Mark,
> >> >> >> >>> > but how about custom vex dso? My func() opens file from
> disk,
> >> but
> >> >> >> >>> > doesn't handle op: syntax. Is it doable in custom code?
> >> >> >> >>> >
> >> >> >> >>> > thanks,
> >> >> >> >>> > skk.
> >> >> >> >>> >
> >> >> >> >>> > 2011/1/19 Mark Elendt <mark at sidefx.com>:
> >> >> >> >>> > > Unfortunately, at the current time op: is handled as a
> >> special
> >> >> >> case,
> >> >> >> >>> > > so it's not part of the generalized loader.
> >> >> >> >>> > >
> >> >> >> >>> > > You'll have to reference the geometry directly (i.e. cook
> the
> >> >> SOP).
> >> >> >> >>> > > If you need your own copy, you'll have to merge it.
> >> >> >> >>> > >
> >> >> >> >>> > > On the other hand, opdef: is handled properly.
> >> >> >> >>> > >
> >> >> >> >>> > > On Wednesday Jan 19 at 12:11, Olex P wrote:
> >> >> >> >>> > >> Maybe you right Szymon. Most of the usage of op: syntax
> for
> >> me
> >> >> was
> >> >> >> >>> sampling
> >> >> >> >>> > >> from point clouds/volumes/textures and I have never tries
> to
> >> >> load
> >> >> >> >>> geometry
> >> >> >> >>> > >> using op: syntax. File SOP doesn't work with it and I
> think
> >> >> it's
> >> >> >> >>> because of
> >> >> >> >>> > >> the fact that load() expects a proper file format. Would
> be
> >> >> great
> >> >> >> to
> >> >> >> >>> hear
> >> >> >> >>> > >> about this from SESI.
> >> >> >> >>> > >>
> >> >> >> >>> > >> On Wed, Jan 19, 2011 at 11:58 AM, Szymon Kapeniak <
> >> >> >> >>> szymon.kapeniak at gmail.com
> >> >> >> >>> > >> > wrote:
> >> >> >> >>> > >>
> >> >> >> >>> > >> > hi Olex,
> >> >> >> >>> > >> > thanks for fast response. Well, I must be doing
> something
> >> >> >> stupid,
> >> >> >> >>> > >> > because it doesn't.
> >> >> >> >>> > >> >
> >> >> >> >>> > >> > GU_Detail gdp;
> >> >> >> >>> > >> > const char *file   = (const char *) argv[1];
> >> >> >> >>> > >> > if (gdp.load(file, 0) < 0) return ;
> >> >> >> >>> > >> > ...
> >> >> >> >>> > >> >
> >> >> >> >>> > >> > while works fine with disk paths, doesn't work with
> >> op:/'ed
> >> >> one.
> >> >> >> >>> > >> >
> >> >> >> >>> > >> >
> >> >> >> >>> > >> > 2011/1/19 Olex P <hoknamahn at gmail.com>:
> >> >> >> >>> > >> > > Hi Szymon,
> >> >> >> >>> > >> > >
> >> >> >> >>> > >> > > I don't think that you need any magic. op: syntax has
> to
> >> be
> >> >> >> >>> handled
> >> >> >> >>> > >> > > automagically when you using
> >> >> >> >>> > >> > >
> >> >> >> >>> > >> > > [code]virtual int          load(const char *, const
> >> >> UT_Options
> >> >> >> >>> > >> > > *options);[/code]
> >> >> >> >>> > >> > >
> >> >> >> >>> > >> > > method.
> >> >> >> >>> > >> > >
> >> >> >> >>> > >> > > On Wed, Jan 19, 2011 at 11:41 AM, Szymon Kapeniak <
> >> >> >> >>> > >> > szymon.kapeniak at gmail.com
> >> >> >> >>> > >> > >> wrote:
> >> >> >> >>> > >> > >
> >> >> >> >>> > >> > >> greetings!
> >> >> >> >>> > >> > >>
> >> >> >> >>> > >> > >> what is a magic to let  my GU_Detail class to
> ::load()
> >> >> both
> >> >> >> files
> >> >> >> >>> from
> >> >> >> >>> > >> > >> disk as operator's streams with op:/path syntax? I
> can
> >> see
> >> >> >> load()
> >> >> >> >>> > >> > >> with an UT_IStream as argument, but I have no idea
> how
> >> to
> >> >> use
> >> >> >> it.
> >> >> >> >>> Does
> >> >> >> >>> > >> > >> any of UT_Stream versions handle both files and
> steams
> >> or
> >> >> the
> >> >> >> >>> logic is
> >> >> >> >>> > >> > >> on my side?
> >> >> >> >>> > >> > >>
> >> >> >> >>> > >> > >> thanks,
> >> >> >> >>> > >> > >> skk.
> >> >> >> >>> > >> > >> _______________________________________________
> >> >> >> >>> > >> > >> Sidefx-houdini-list mailing list
> >> >> >> >>> > >> > >> Sidefx-houdini-list at sidefx.com
> >> >> >> >>> > >> > >>
> >> >> >> >>>
> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >> >>> > >> > >>
> >> >> >> >>> > >> > > _______________________________________________
> >> >> >> >>> > >> > > Sidefx-houdini-list mailing list
> >> >> >> >>> > >> > > Sidefx-houdini-list at sidefx.com
> >> >> >> >>> > >> > >
> >> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >> >>> > >> > >
> >> >> >> >>> > >> > _______________________________________________
> >> >> >> >>> > >> > Sidefx-houdini-list mailing list
> >> >> >> >>> > >> > Sidefx-houdini-list at sidefx.com
> >> >> >> >>> > >> >
> >> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >> >>> > >> >
> >> >> >> >>> > >> _______________________________________________
> >> >> >> >>> > >> Sidefx-houdini-list mailing list
> >> >> >> >>> > >> Sidefx-houdini-list at sidefx.com
> >> >> >> >>> > >>
> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >> >>> > >
> >> >> >> >>> > > --
> >> >> >> >>> > > Mark Elendt
> >> >> >> >>> > > _______________________________________________
> >> >> >> >>> > > Sidefx-houdini-list mailing list
> >> >> >> >>> > > Sidefx-houdini-list at sidefx.com
> >> >> >> >>> > >
> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >> >>> > >
> >> >> >> >>> > _______________________________________________
> >> >> >> >>> > Sidefx-houdini-list mailing list
> >> >> >> >>> > Sidefx-houdini-list at sidefx.com
> >> >> >> >>> >
> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> Mark Elendt
> >> >> >> >>> _______________________________________________
> >> >> >> >>> Sidefx-houdini-list mailing list
> >> >> >> >>> Sidefx-houdini-list at sidefx.com
> >> >> >> >>>
> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >> >>>
> >> >> >> >> _______________________________________________
> >> >> >> >> Sidefx-houdini-list mailing list
> >> >> >> >> Sidefx-houdini-list at sidefx.com
> >> >> >> >>
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >> >>
> >> >> >> >
> >> >> >> _______________________________________________
> >> >> >> Sidefx-houdini-list mailing list
> >> >> >> Sidefx-houdini-list at sidefx.com
> >> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >>
> >> >> > _______________________________________________
> >> >> > Sidefx-houdini-list mailing list
> >> >> > Sidefx-houdini-list at sidefx.com
> >> >> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >> >
> >> >> _______________________________________________
> >> >> Sidefx-houdini-list mailing list
> >> >> Sidefx-houdini-list at sidefx.com
> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >>
> >> > _______________________________________________
> >> > Sidefx-houdini-list mailing list
> >> > Sidefx-houdini-list at sidefx.com
> >> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >> >
> >> _______________________________________________
> >> Sidefx-houdini-list mailing list
> >> Sidefx-houdini-list at sidefx.com
> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>
> > _______________________________________________
> > Sidefx-houdini-list mailing list
> > Sidefx-houdini-list at sidefx.com
> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >
> _______________________________________________
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>



More information about the Sidefx-houdini-list mailing list