[Sidefx-houdini-list] HDK: op:/ magic

Olex P hoknamahn at gmail.com
Sat Jan 22 17:54:22 EST 2011


Hi Szymon,

The answer is simple: use a counter of threads running. Each thread
increments this counter when it starts and decrements when it finishes. And
as you know a total number of threads used you can find out which thread is
last. That thread kills an object. And loading is even simpler: if(geo /*
i.e. if geo != NULL */) load()

On Sat, Jan 22, 2011 at 10:39 PM, Szymon Kapeniak <szymon.kapeniak at gmail.com
> wrote:

> Hi Olex,
> hmm... I'm doing pretty much the same things, even checking thread ids ;).
>
> I made some more tests, and I seem to be seeing a root of evil. My
> wrong assumption was that init() and cleanup() are called at the start
> & end by the main thread, which is wrong, as they are called by every
> thread - which means I can't delete gdp unconditionally in cleanup,
> without pissing off still running threads ;)
>
> Now I need to find out how to load only once and delete gdp in a last
> running thread. I'm trying reference counter with no success at the
> moment...
>
> thanks again for your help.
>
> skk.
>
>
> 2011/1/22 Olex P <hoknamahn at gmail.com>:
> > Well this bit of code
> >
> > cout << "Process id = " << UT_Thread::getMyThreadId() << endl;
> >
> > gave me different ids so I assume that VEX op was running on several
> > threads. I hope :)
> >
> > On Sat, Jan 22, 2011 at 1:42 AM, Olex P <hoknamahn at gmail.com> wrote:
> >
> >> Yes, test was heavy enough (1024x1024). So in theory it had to be
> running
> >> on 8 threads. But top gave me 15 to 90 percent of CPU used so I'm not
> sure
> >> was it really multithreaded. Pprobably it makes sense to do one more
> test
> >> that gives you the id of a thread it's running on.
> >> GDP object was allocated in Init(). Same with loading of geometry.
> >> I think you can also do first loading in Evaluate() in order to be able
> to
> >> use the file name passed to VEX op. Shouldn't be a problem. I'll check
> that
> >> tomorrow.
> >>
> >>
> >> On Sat, Jan 22, 2011 at 12:17 AM, Szymon Kapeniak <
> >> szymon.kapeniak at gmail.com> wrote:
> >>
> >>> That would be fantastic! But is your geometry enough heavy to start
> >>> multi-threading? Grid 100x100 should be fine, but anything smaller
> >>> doesn't use threads even if you set it on a vex node.
> >>>
> >>> Secondly, do you allocate new GU_Detail and load bgeo inside init()? I
> >>> would love to do that, but I don't know how to pass user arguments
> >>> there. It doesn't seem to take any, so all I have must take place
> >>> inside main function. I may be wrong also in this point though...
> >>>
> >>> I'll try your example tomorrow.
> >>>
> >>> Thank you Olex!
> >>>
> >>> 2011/1/22 Olex P <hoknamahn at gmail.com>:
> >>> > Hi Szymon,
> >>> >
> >>> > I've made a simple VEX dso which creates a static pointer to
> GU_Detail
> >>> >
> >>> > static GU_Detail* geo = NULL;
> >>> >
> >>> > allocates new object on Init()
> >>> >
> >>> > if(!geo)
> >>> > {
> >>> >  geo = new GU_Detail();
> >>> >  geo->load("/tmp/something.bgeo", 0);
> >>> > }
> >>> >
> >>> > takes an attribute value on Evaluate and returns it
> >>> >
> >>> > GEO_Point* pt = geo->points()[ptnum];
> >>> > GB_AttributeRef somethingIndex = geo->findPointAttrib("something",
> >>> > GB_ATTRIB_FLOAT);
> >>> > const float* something = pt->castAttribData<float>(somethingIndex);
> >>> > *result = *something;
> >>> >
> >>> > VEX op itself looks like:
> >>> >
> >>> > sop
> >>> > vextest(export float newsomething = 0)
> >>> > {
> >>> >    newsomething = myfun((float)ptnum);
> >>> > }
> >>> >
> >>> > So we create a geometry, apply an attribute on it, save somewhere.
> Then
> >>> > apply this VEX op to clean (without an attribute geo) and spreadsheet
> >>> show
> >>> > exactly the same values as where written into bgeo.
> >>> > It works fine.
> >>> > On Fri, Jan 21, 2011 at 6:30 PM, Szymon Kapeniak
> >>> > <szymon.kapeniak at gmail.com>wrote:
> >>> >
> >>> >> The problem is I don't have a chance to access any attribute, read,
> >>> >> write, what so ever. I declare gdp as static pointer and create new
> >>> >> GU_Detail inside init(), which meant to run once before threads
> start,
> >>> >> then I point threads to that gdp as advised in docs. It's enough to
> >>> >> crash Houdini in sop context.
> >>> >>
> >>> >> 2011/1/21 Olex P <hoknamahn at gmail.com>:
> >>> >> > Hi Szymon,
> >>> >> >
> >>> >> > VEX itself has nothing to do with multithreading. But VEX engine
> >>> >> (standard
> >>> >> > contexts or your own CVEX one) does. So in the case you mentioned
> >>> (VEX
> >>> >> SOP
> >>> >> > that is reading a point cloud and assigning a filtered value to a
> >>> point)
> >>> >> it
> >>> >> > works as (simplified):
> >>> >> > 1. VEX engine (SOP context) creates a flat array and fills it in
> with
> >>> the
> >>> >> > attributes from geometry
> >>> >> > 2. VEX engine spawns several threads and runs one instance of VEX
> >>> >> procedure
> >>> >> > in each thread
> >>> >> > 3. Each instance of VEX procedure (single threaded) filters a
> value
> >>> from
> >>> >> > point cloud and writes into it's chunk of flat array
> >>> >> > 4. VEX engine writes all the values back from flat array into a
> GDP
> >>> using
> >>> >> > one thread
> >>> >> > Honestly I haven't tried multithreading with GDP but I see no
> >>> problems
> >>> >> with
> >>> >> > it as long as you are not modifying the data. And normally VEX
> >>> function
> >>> >> > itself doesn't modify a GDP (but VEX engine does).
> >>> >> > Well that's how I understand it and I can be wrong :)
> >>> >> > There are some bits about thread safety in HDK
> >>> >> >
> >>> >>
> >>>
> http://www.sidefx.com/docs/hdk11.0/hdk_changes_11_0.html#HDKC11_0_ThreadSafety
> >>> >> > I probably have to do some tests too :)
> >>> >> >
> >>> >> > On Fri, Jan 21, 2011 at 5:13 PM, Szymon Kapeniak
> >>> >> > <szymon.kapeniak at gmail.com>wrote:
> >>> >> >
> >>> >> >> Hey Olex,
> >>> >> >> I'm writing custom vex, and I'm sure vex has some standard
> methods
> >>> to
> >>> >> >> deal with this. I'm only interested in read access from several
> >>> >> >> threads which I don't have any control on, as they are spawned by
> >>> VEX
> >>> >> >> SOP.
> >>> >> >>
> >>> >> >> I started to play with it by checking minimal scenarios and it
> seems
> >>> >> >> it's enough to declare one static GU_Detail instance and assign
> >>> >> >> pointers to it from a couple of threads to crash Houdini even
> >>> without
> >>> >> >> loading any geometry.
> >>> >> >> Also by checking threads ids I realized, that mantra can run the
> >>> same
> >>> >> >> code multi-threaded because vex in surface context call init()
> >>> >> >> (standard procedure) per thread, unlike in sop context, where it
> >>> seems
> >>> >> >> init() is called only once like in fact it's described in docs.
> >>> >> >>
> >>> >> >> Surly there are thread-save vex functions (reading gdp) which
> >>> operate
> >>> >> >> in sop context, like pcopen().  I'm guessing the trick is not to
> use
> >>> >> >> pure gdp, but some  thread-safe memory schema. Lack of
> documentation
> >>> >> >> doesn't play well with my ignorance...
> >>> >> >>
> >>> >> >> skk.
> >>> >> >>
> >>> >> >> btw  I'm referring to this talk:
> >>> >> >>
> >>> >> >>
> >>> >>
> >>>
> http://forums.odforce.net/index.php?/topic/12434-how-to-replicate-this-ice-network-in-vops/page__st__12
> >>> >> >>
> >>> >> >> 2011/1/21 Olex P <hoknamahn at gmail.com>:
> >>> >> >> > Hey Szymon,
> >>> >> >> >
> >>> >> >> > You can read in multiple threads but multithread writing as far
> as
> >>> I
> >>> >> know
> >>> >> >> is
> >>> >> >> > limited to volume primitives (each thread can write into it's
> own
> >>> >> tile)
> >>> >> >> as
> >>> >> >> > far as I know. You could create own GU/GEO_Detail object for
> each
> >>> >> thread,
> >>> >> >> > write into it and then merge all the pieces of puzzle into one
> >>> object
> >>> >> >> (merge
> >>> >> >> > will single thread).
> >>> >> >> > But from your words I'm not sure what kind of parallelism you
> are
> >>> >> looking
> >>> >> >> > for...
> >>> >> >> > Let's say sampling from volume in VEX. 1. You are running the
> same
> >>> >> single
> >>> >> >> > thread code on different chunks of data using multiple threads.
> 2.
> >>> But
> >>> >> >> this
> >>> >> >> > also could be done in a different way: vex function internally
> >>> spawns
> >>> >> >> > several threads which modify the data somehow. You probably
> >>> talking
> >>> >> about
> >>> >> >> 2
> >>> >> >> > (as you mentioned multithread access to GU_Detail). I'm not
> sure
> >>> but
> >>> >> >> maybe
> >>> >> >> > it makes sense to use 1 (i.e. CVEX). Sure it will work only for
> >>> data
> >>> >> >> > parallel algorithms.
> >>> >> >> >
> >>> >> >> > On Thu, Jan 20, 2011 at 10:20 PM, Szymon Kapeniak <
> >>> >> >> szymon.kapeniak at gmail.com
> >>> >> >> >> wrote:
> >>> >> >> >
> >>> >> >> >> ...sorry for spamming, but I've just realized that the same
> dso
> >>> works
> >>> >> >> >> in mantra, but not in (threading) sop context. Kind of
> strange.
> >>> >> >> >>
> >>> >> >> >> 2011/1/20 Szymon Kapeniak <szymon.kapeniak at gmail.com>:
> >>> >> >> >> > Thanks for a response, I didn't know I have to deal with it
> >>> >> manually.
> >>> >> >> >> >
> >>> >> >> >> > Second problem is threading in vex sop. After playing with
> my
> >>> >> function
> >>> >> >> >> > a little, I'm getting a feeling my approach was too naive,
> so
> >>> to
> >>> >> >> >> > speak.
> >>> >> >> >> >
> >>> >> >> >> > Is multi-threaded access to a static GU_Detail * possible at
> >>> all,
> >>> >> or
> >>> >> >> >> > do I have to use some caches like UT_ThreadSafeCache or
> >>> anything
> >>> >> >> >> > similar? What is a general way of doing this in HDK?
> >>> >> >> >> >
> >>> >> >> >> > thanks,
> >>> >> >> >> > skk.
> >>> >> >> >> >
> >>> >> >> >> >
> >>> >> >> >> > 2011/1/19 Olex P <hoknamahn at gmail.com>:
> >>> >> >> >> >> It kind of does writing an op-referenced texture into an
> IFD
> >>> :)
> >>> >> >> >> >>
> >>> >> >> >> >> On Wed, Jan 19, 2011 at 1:48 PM, Mark Elendt <
> mark at sidefx.com
> >>> >
> >>> >> >> wrote:
> >>> >> >> >> >>
> >>> >> >> >> >>> For mantra, even Houdini's VEX doesn't support the op:
> syntax
> >>> :)
> >>> >> >> >> >>>
> >>> >> >> >> >>> For VEX functions in OPs, you should be able to access the
> >>> SOP by
> >>> >> >> >> >>> grabbing the OP director and finding the SOP.
> >>> >> >> >> >>>
> >>> >> >> >> >>> On Wednesday Jan 19 at 13:39, Szymon Kapeniak wrote:
> >>> >> >> >> >>> > thanks Mark,
> >>> >> >> >> >>> > but how about custom vex dso? My func() opens file from
> >>> disk,
> >>> >> but
> >>> >> >> >> >>> > doesn't handle op: syntax. Is it doable in custom code?
> >>> >> >> >> >>> >
> >>> >> >> >> >>> > thanks,
> >>> >> >> >> >>> > skk.
> >>> >> >> >> >>> >
> >>> >> >> >> >>> > 2011/1/19 Mark Elendt <mark at sidefx.com>:
> >>> >> >> >> >>> > > Unfortunately, at the current time op: is handled as a
> >>> >> special
> >>> >> >> >> case,
> >>> >> >> >> >>> > > so it's not part of the generalized loader.
> >>> >> >> >> >>> > >
> >>> >> >> >> >>> > > You'll have to reference the geometry directly (i.e.
> cook
> >>> the
> >>> >> >> SOP).
> >>> >> >> >> >>> > > If you need your own copy, you'll have to merge it.
> >>> >> >> >> >>> > >
> >>> >> >> >> >>> > > On the other hand, opdef: is handled properly.
> >>> >> >> >> >>> > >
> >>> >> >> >> >>> > > On Wednesday Jan 19 at 12:11, Olex P wrote:
> >>> >> >> >> >>> > >> Maybe you right Szymon. Most of the usage of op:
> syntax
> >>> for
> >>> >> me
> >>> >> >> was
> >>> >> >> >> >>> sampling
> >>> >> >> >> >>> > >> from point clouds/volumes/textures and I have never
> >>> tries to
> >>> >> >> load
> >>> >> >> >> >>> geometry
> >>> >> >> >> >>> > >> using op: syntax. File SOP doesn't work with it and I
> >>> think
> >>> >> >> it's
> >>> >> >> >> >>> because of
> >>> >> >> >> >>> > >> the fact that load() expects a proper file format.
> Would
> >>> be
> >>> >> >> great
> >>> >> >> >> to
> >>> >> >> >> >>> hear
> >>> >> >> >> >>> > >> about this from SESI.
> >>> >> >> >> >>> > >>
> >>> >> >> >> >>> > >> On Wed, Jan 19, 2011 at 11:58 AM, Szymon Kapeniak <
> >>> >> >> >> >>> szymon.kapeniak at gmail.com
> >>> >> >> >> >>> > >> > wrote:
> >>> >> >> >> >>> > >>
> >>> >> >> >> >>> > >> > hi Olex,
> >>> >> >> >> >>> > >> > thanks for fast response. Well, I must be doing
> >>> something
> >>> >> >> >> stupid,
> >>> >> >> >> >>> > >> > because it doesn't.
> >>> >> >> >> >>> > >> >
> >>> >> >> >> >>> > >> > GU_Detail gdp;
> >>> >> >> >> >>> > >> > const char *file   = (const char *) argv[1];
> >>> >> >> >> >>> > >> > if (gdp.load(file, 0) < 0) return ;
> >>> >> >> >> >>> > >> > ...
> >>> >> >> >> >>> > >> >
> >>> >> >> >> >>> > >> > while works fine with disk paths, doesn't work with
> >>> >> op:/'ed
> >>> >> >> one.
> >>> >> >> >> >>> > >> >
> >>> >> >> >> >>> > >> >
> >>> >> >> >> >>> > >> > 2011/1/19 Olex P <hoknamahn at gmail.com>:
> >>> >> >> >> >>> > >> > > Hi Szymon,
> >>> >> >> >> >>> > >> > >
> >>> >> >> >> >>> > >> > > I don't think that you need any magic. op: syntax
> >>> has to
> >>> >> be
> >>> >> >> >> >>> handled
> >>> >> >> >> >>> > >> > > automagically when you using
> >>> >> >> >> >>> > >> > >
> >>> >> >> >> >>> > >> > > [code]virtual int          load(const char *,
> const
> >>> >> >> UT_Options
> >>> >> >> >> >>> > >> > > *options);[/code]
> >>> >> >> >> >>> > >> > >
> >>> >> >> >> >>> > >> > > method.
> >>> >> >> >> >>> > >> > >
> >>> >> >> >> >>> > >> > > On Wed, Jan 19, 2011 at 11:41 AM, Szymon Kapeniak
> <
> >>> >> >> >> >>> > >> > szymon.kapeniak at gmail.com
> >>> >> >> >> >>> > >> > >> wrote:
> >>> >> >> >> >>> > >> > >
> >>> >> >> >> >>> > >> > >> greetings!
> >>> >> >> >> >>> > >> > >>
> >>> >> >> >> >>> > >> > >> what is a magic to let  my GU_Detail class to
> >>> ::load()
> >>> >> >> both
> >>> >> >> >> files
> >>> >> >> >> >>> from
> >>> >> >> >> >>> > >> > >> disk as operator's streams with op:/path syntax?
> I
> >>> can
> >>> >> see
> >>> >> >> >> load()
> >>> >> >> >> >>> > >> > >> with an UT_IStream as argument, but I have no
> idea
> >>> how
> >>> >> to
> >>> >> >> use
> >>> >> >> >> it.
> >>> >> >> >> >>> Does
> >>> >> >> >> >>> > >> > >> any of UT_Stream versions handle both files and
> >>> steams
> >>> >> or
> >>> >> >> the
> >>> >> >> >> >>> logic is
> >>> >> >> >> >>> > >> > >> on my side?
> >>> >> >> >> >>> > >> > >>
> >>> >> >> >> >>> > >> > >> thanks,
> >>> >> >> >> >>> > >> > >> skk.
> >>> >> >> >> >>> > >> > >> _______________________________________________
> >>> >> >> >> >>> > >> > >> Sidefx-houdini-list mailing list
> >>> >> >> >> >>> > >> > >> Sidefx-houdini-list at sidefx.com
> >>> >> >> >> >>> > >> > >>
> >>> >> >> >> >>>
> >>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >> >>> > >> > >>
> >>> >> >> >> >>> > >> > > _______________________________________________
> >>> >> >> >> >>> > >> > > Sidefx-houdini-list mailing list
> >>> >> >> >> >>> > >> > > Sidefx-houdini-list at sidefx.com
> >>> >> >> >> >>> > >> > >
> >>> >> >> >>
> >>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >> >>> > >> > >
> >>> >> >> >> >>> > >> > _______________________________________________
> >>> >> >> >> >>> > >> > Sidefx-houdini-list mailing list
> >>> >> >> >> >>> > >> > Sidefx-houdini-list at sidefx.com
> >>> >> >> >> >>> > >> >
> >>> >> >> >>
> >>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >> >>> > >> >
> >>> >> >> >> >>> > >> _______________________________________________
> >>> >> >> >> >>> > >> Sidefx-houdini-list mailing list
> >>> >> >> >> >>> > >> Sidefx-houdini-list at sidefx.com
> >>> >> >> >> >>> > >>
> >>> >> >>
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >> >>> > >
> >>> >> >> >> >>> > > --
> >>> >> >> >> >>> > > Mark Elendt
> >>> >> >> >> >>> > > _______________________________________________
> >>> >> >> >> >>> > > Sidefx-houdini-list mailing list
> >>> >> >> >> >>> > > Sidefx-houdini-list at sidefx.com
> >>> >> >> >> >>> > >
> >>> >> >>
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >> >>> > >
> >>> >> >> >> >>> > _______________________________________________
> >>> >> >> >> >>> > Sidefx-houdini-list mailing list
> >>> >> >> >> >>> > Sidefx-houdini-list at sidefx.com
> >>> >> >> >> >>> >
> >>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >> >>>
> >>> >> >> >> >>> --
> >>> >> >> >> >>> Mark Elendt
> >>> >> >> >> >>> _______________________________________________
> >>> >> >> >> >>> Sidefx-houdini-list mailing list
> >>> >> >> >> >>> Sidefx-houdini-list at sidefx.com
> >>> >> >> >> >>>
> >>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >> >>>
> >>> >> >> >> >> _______________________________________________
> >>> >> >> >> >> Sidefx-houdini-list mailing list
> >>> >> >> >> >> Sidefx-houdini-list at sidefx.com
> >>> >> >> >> >>
> >>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >> >>
> >>> >> >> >> >
> >>> >> >> >> _______________________________________________
> >>> >> >> >> Sidefx-houdini-list mailing list
> >>> >> >> >> Sidefx-houdini-list at sidefx.com
> >>> >> >> >>
> >>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >>
> >>> >> >> > _______________________________________________
> >>> >> >> > Sidefx-houdini-list mailing list
> >>> >> >> > Sidefx-houdini-list at sidefx.com
> >>> >> >> >
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >> >
> >>> >> >> _______________________________________________
> >>> >> >> Sidefx-houdini-list mailing list
> >>> >> >> Sidefx-houdini-list at sidefx.com
> >>> >> >>
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >>
> >>> >> > _______________________________________________
> >>> >> > Sidefx-houdini-list mailing list
> >>> >> > Sidefx-houdini-list at sidefx.com
> >>> >> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >> >
> >>> >> _______________________________________________
> >>> >> Sidefx-houdini-list mailing list
> >>> >> Sidefx-houdini-list at sidefx.com
> >>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >>
> >>> > _______________________________________________
> >>> > Sidefx-houdini-list mailing list
> >>> > Sidefx-houdini-list at sidefx.com
> >>> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>> >
> >>> _______________________________________________
> >>> Sidefx-houdini-list mailing list
> >>> Sidefx-houdini-list at sidefx.com
> >>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >>>
> >>
> >>
> > _______________________________________________
> > Sidefx-houdini-list mailing list
> > Sidefx-houdini-list at sidefx.com
> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
> >
> _______________________________________________
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>



More information about the Sidefx-houdini-list mailing list