[Sidefx-houdini-list] HDK: op:/ magic

Szymon Kapeniak szymon.kapeniak at gmail.com
Sat Jan 22 17:39:25 EST 2011


Hi Olex,
hmm... I'm doing pretty much the same things, even checking thread ids ;).

I made some more tests, and I seem to be seeing a root of evil. My
wrong assumption was that init() and cleanup() are called at the start
& end by the main thread, which is wrong, as they are called by every
thread - which means I can't delete gdp unconditionally in cleanup,
without pissing off still running threads ;)

Now I need to find out how to load only once and delete gdp in a last
running thread. I'm trying reference counter with no success at the
moment...

thanks again for your help.

skk.


2011/1/22 Olex P <hoknamahn at gmail.com>:
> Well this bit of code
>
> cout << "Process id = " << UT_Thread::getMyThreadId() << endl;
>
> gave me different ids so I assume that VEX op was running on several
> threads. I hope :)
>
> On Sat, Jan 22, 2011 at 1:42 AM, Olex P <hoknamahn at gmail.com> wrote:
>
>> Yes, test was heavy enough (1024x1024). So in theory it had to be running
>> on 8 threads. But top gave me 15 to 90 percent of CPU used so I'm not sure
>> was it really multithreaded. Pprobably it makes sense to do one more test
>> that gives you the id of a thread it's running on.
>> GDP object was allocated in Init(). Same with loading of geometry.
>> I think you can also do first loading in Evaluate() in order to be able to
>> use the file name passed to VEX op. Shouldn't be a problem. I'll check that
>> tomorrow.
>>
>>
>> On Sat, Jan 22, 2011 at 12:17 AM, Szymon Kapeniak <
>> szymon.kapeniak at gmail.com> wrote:
>>
>>> That would be fantastic! But is your geometry enough heavy to start
>>> multi-threading? Grid 100x100 should be fine, but anything smaller
>>> doesn't use threads even if you set it on a vex node.
>>>
>>> Secondly, do you allocate new GU_Detail and load bgeo inside init()? I
>>> would love to do that, but I don't know how to pass user arguments
>>> there. It doesn't seem to take any, so all I have must take place
>>> inside main function. I may be wrong also in this point though...
>>>
>>> I'll try your example tomorrow.
>>>
>>> Thank you Olex!
>>>
>>> 2011/1/22 Olex P <hoknamahn at gmail.com>:
>>> > Hi Szymon,
>>> >
>>> > I've made a simple VEX dso which creates a static pointer to GU_Detail
>>> >
>>> > static GU_Detail* geo = NULL;
>>> >
>>> > allocates new object on Init()
>>> >
>>> > if(!geo)
>>> > {
>>> >  geo = new GU_Detail();
>>> >  geo->load("/tmp/something.bgeo", 0);
>>> > }
>>> >
>>> > takes an attribute value on Evaluate and returns it
>>> >
>>> > GEO_Point* pt = geo->points()[ptnum];
>>> > GB_AttributeRef somethingIndex = geo->findPointAttrib("something",
>>> > GB_ATTRIB_FLOAT);
>>> > const float* something = pt->castAttribData<float>(somethingIndex);
>>> > *result = *something;
>>> >
>>> > VEX op itself looks like:
>>> >
>>> > sop
>>> > vextest(export float newsomething = 0)
>>> > {
>>> >    newsomething = myfun((float)ptnum);
>>> > }
>>> >
>>> > So we create a geometry, apply an attribute on it, save somewhere. Then
>>> > apply this VEX op to clean (without an attribute geo) and spreadsheet
>>> show
>>> > exactly the same values as where written into bgeo.
>>> > It works fine.
>>> > On Fri, Jan 21, 2011 at 6:30 PM, Szymon Kapeniak
>>> > <szymon.kapeniak at gmail.com>wrote:
>>> >
>>> >> The problem is I don't have a chance to access any attribute, read,
>>> >> write, what so ever. I declare gdp as static pointer and create new
>>> >> GU_Detail inside init(), which meant to run once before threads start,
>>> >> then I point threads to that gdp as advised in docs. It's enough to
>>> >> crash Houdini in sop context.
>>> >>
>>> >> 2011/1/21 Olex P <hoknamahn at gmail.com>:
>>> >> > Hi Szymon,
>>> >> >
>>> >> > VEX itself has nothing to do with multithreading. But VEX engine
>>> >> (standard
>>> >> > contexts or your own CVEX one) does. So in the case you mentioned
>>> (VEX
>>> >> SOP
>>> >> > that is reading a point cloud and assigning a filtered value to a
>>> point)
>>> >> it
>>> >> > works as (simplified):
>>> >> > 1. VEX engine (SOP context) creates a flat array and fills it in with
>>> the
>>> >> > attributes from geometry
>>> >> > 2. VEX engine spawns several threads and runs one instance of VEX
>>> >> procedure
>>> >> > in each thread
>>> >> > 3. Each instance of VEX procedure (single threaded) filters a value
>>> from
>>> >> > point cloud and writes into it's chunk of flat array
>>> >> > 4. VEX engine writes all the values back from flat array into a GDP
>>> using
>>> >> > one thread
>>> >> > Honestly I haven't tried multithreading with GDP but I see no
>>> problems
>>> >> with
>>> >> > it as long as you are not modifying the data. And normally VEX
>>> function
>>> >> > itself doesn't modify a GDP (but VEX engine does).
>>> >> > Well that's how I understand it and I can be wrong :)
>>> >> > There are some bits about thread safety in HDK
>>> >> >
>>> >>
>>> http://www.sidefx.com/docs/hdk11.0/hdk_changes_11_0.html#HDKC11_0_ThreadSafety
>>> >> > I probably have to do some tests too :)
>>> >> >
>>> >> > On Fri, Jan 21, 2011 at 5:13 PM, Szymon Kapeniak
>>> >> > <szymon.kapeniak at gmail.com>wrote:
>>> >> >
>>> >> >> Hey Olex,
>>> >> >> I'm writing custom vex, and I'm sure vex has some standard methods
>>> to
>>> >> >> deal with this. I'm only interested in read access from several
>>> >> >> threads which I don't have any control on, as they are spawned by
>>> VEX
>>> >> >> SOP.
>>> >> >>
>>> >> >> I started to play with it by checking minimal scenarios and it seems
>>> >> >> it's enough to declare one static GU_Detail instance and assign
>>> >> >> pointers to it from a couple of threads to crash Houdini even
>>> without
>>> >> >> loading any geometry.
>>> >> >> Also by checking threads ids I realized, that mantra can run the
>>> same
>>> >> >> code multi-threaded because vex in surface context call init()
>>> >> >> (standard procedure) per thread, unlike in sop context, where it
>>> seems
>>> >> >> init() is called only once like in fact it's described in docs.
>>> >> >>
>>> >> >> Surly there are thread-save vex functions (reading gdp) which
>>> operate
>>> >> >> in sop context, like pcopen().  I'm guessing the trick is not to use
>>> >> >> pure gdp, but some  thread-safe memory schema. Lack of documentation
>>> >> >> doesn't play well with my ignorance...
>>> >> >>
>>> >> >> skk.
>>> >> >>
>>> >> >> btw  I'm referring to this talk:
>>> >> >>
>>> >> >>
>>> >>
>>> http://forums.odforce.net/index.php?/topic/12434-how-to-replicate-this-ice-network-in-vops/page__st__12
>>> >> >>
>>> >> >> 2011/1/21 Olex P <hoknamahn at gmail.com>:
>>> >> >> > Hey Szymon,
>>> >> >> >
>>> >> >> > You can read in multiple threads but multithread writing as far as
>>> I
>>> >> know
>>> >> >> is
>>> >> >> > limited to volume primitives (each thread can write into it's own
>>> >> tile)
>>> >> >> as
>>> >> >> > far as I know. You could create own GU/GEO_Detail object for each
>>> >> thread,
>>> >> >> > write into it and then merge all the pieces of puzzle into one
>>> object
>>> >> >> (merge
>>> >> >> > will single thread).
>>> >> >> > But from your words I'm not sure what kind of parallelism you are
>>> >> looking
>>> >> >> > for...
>>> >> >> > Let's say sampling from volume in VEX. 1. You are running the same
>>> >> single
>>> >> >> > thread code on different chunks of data using multiple threads. 2.
>>> But
>>> >> >> this
>>> >> >> > also could be done in a different way: vex function internally
>>> spawns
>>> >> >> > several threads which modify the data somehow. You probably
>>> talking
>>> >> about
>>> >> >> 2
>>> >> >> > (as you mentioned multithread access to GU_Detail). I'm not sure
>>> but
>>> >> >> maybe
>>> >> >> > it makes sense to use 1 (i.e. CVEX). Sure it will work only for
>>> data
>>> >> >> > parallel algorithms.
>>> >> >> >
>>> >> >> > On Thu, Jan 20, 2011 at 10:20 PM, Szymon Kapeniak <
>>> >> >> szymon.kapeniak at gmail.com
>>> >> >> >> wrote:
>>> >> >> >
>>> >> >> >> ...sorry for spamming, but I've just realized that the same dso
>>> works
>>> >> >> >> in mantra, but not in (threading) sop context. Kind of strange.
>>> >> >> >>
>>> >> >> >> 2011/1/20 Szymon Kapeniak <szymon.kapeniak at gmail.com>:
>>> >> >> >> > Thanks for a response, I didn't know I have to deal with it
>>> >> manually.
>>> >> >> >> >
>>> >> >> >> > Second problem is threading in vex sop. After playing with my
>>> >> function
>>> >> >> >> > a little, I'm getting a feeling my approach was too naive, so
>>> to
>>> >> >> >> > speak.
>>> >> >> >> >
>>> >> >> >> > Is multi-threaded access to a static GU_Detail * possible at
>>> all,
>>> >> or
>>> >> >> >> > do I have to use some caches like UT_ThreadSafeCache or
>>> anything
>>> >> >> >> > similar? What is a general way of doing this in HDK?
>>> >> >> >> >
>>> >> >> >> > thanks,
>>> >> >> >> > skk.
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > 2011/1/19 Olex P <hoknamahn at gmail.com>:
>>> >> >> >> >> It kind of does writing an op-referenced texture into an IFD
>>> :)
>>> >> >> >> >>
>>> >> >> >> >> On Wed, Jan 19, 2011 at 1:48 PM, Mark Elendt <mark at sidefx.com
>>> >
>>> >> >> wrote:
>>> >> >> >> >>
>>> >> >> >> >>> For mantra, even Houdini's VEX doesn't support the op: syntax
>>> :)
>>> >> >> >> >>>
>>> >> >> >> >>> For VEX functions in OPs, you should be able to access the
>>> SOP by
>>> >> >> >> >>> grabbing the OP director and finding the SOP.
>>> >> >> >> >>>
>>> >> >> >> >>> On Wednesday Jan 19 at 13:39, Szymon Kapeniak wrote:
>>> >> >> >> >>> > thanks Mark,
>>> >> >> >> >>> > but how about custom vex dso? My func() opens file from
>>> disk,
>>> >> but
>>> >> >> >> >>> > doesn't handle op: syntax. Is it doable in custom code?
>>> >> >> >> >>> >
>>> >> >> >> >>> > thanks,
>>> >> >> >> >>> > skk.
>>> >> >> >> >>> >
>>> >> >> >> >>> > 2011/1/19 Mark Elendt <mark at sidefx.com>:
>>> >> >> >> >>> > > Unfortunately, at the current time op: is handled as a
>>> >> special
>>> >> >> >> case,
>>> >> >> >> >>> > > so it's not part of the generalized loader.
>>> >> >> >> >>> > >
>>> >> >> >> >>> > > You'll have to reference the geometry directly (i.e. cook
>>> the
>>> >> >> SOP).
>>> >> >> >> >>> > > If you need your own copy, you'll have to merge it.
>>> >> >> >> >>> > >
>>> >> >> >> >>> > > On the other hand, opdef: is handled properly.
>>> >> >> >> >>> > >
>>> >> >> >> >>> > > On Wednesday Jan 19 at 12:11, Olex P wrote:
>>> >> >> >> >>> > >> Maybe you right Szymon. Most of the usage of op: syntax
>>> for
>>> >> me
>>> >> >> was
>>> >> >> >> >>> sampling
>>> >> >> >> >>> > >> from point clouds/volumes/textures and I have never
>>> tries to
>>> >> >> load
>>> >> >> >> >>> geometry
>>> >> >> >> >>> > >> using op: syntax. File SOP doesn't work with it and I
>>> think
>>> >> >> it's
>>> >> >> >> >>> because of
>>> >> >> >> >>> > >> the fact that load() expects a proper file format. Would
>>> be
>>> >> >> great
>>> >> >> >> to
>>> >> >> >> >>> hear
>>> >> >> >> >>> > >> about this from SESI.
>>> >> >> >> >>> > >>
>>> >> >> >> >>> > >> On Wed, Jan 19, 2011 at 11:58 AM, Szymon Kapeniak <
>>> >> >> >> >>> szymon.kapeniak at gmail.com
>>> >> >> >> >>> > >> > wrote:
>>> >> >> >> >>> > >>
>>> >> >> >> >>> > >> > hi Olex,
>>> >> >> >> >>> > >> > thanks for fast response. Well, I must be doing
>>> something
>>> >> >> >> stupid,
>>> >> >> >> >>> > >> > because it doesn't.
>>> >> >> >> >>> > >> >
>>> >> >> >> >>> > >> > GU_Detail gdp;
>>> >> >> >> >>> > >> > const char *file   = (const char *) argv[1];
>>> >> >> >> >>> > >> > if (gdp.load(file, 0) < 0) return ;
>>> >> >> >> >>> > >> > ...
>>> >> >> >> >>> > >> >
>>> >> >> >> >>> > >> > while works fine with disk paths, doesn't work with
>>> >> op:/'ed
>>> >> >> one.
>>> >> >> >> >>> > >> >
>>> >> >> >> >>> > >> >
>>> >> >> >> >>> > >> > 2011/1/19 Olex P <hoknamahn at gmail.com>:
>>> >> >> >> >>> > >> > > Hi Szymon,
>>> >> >> >> >>> > >> > >
>>> >> >> >> >>> > >> > > I don't think that you need any magic. op: syntax
>>> has to
>>> >> be
>>> >> >> >> >>> handled
>>> >> >> >> >>> > >> > > automagically when you using
>>> >> >> >> >>> > >> > >
>>> >> >> >> >>> > >> > > [code]virtual int          load(const char *, const
>>> >> >> UT_Options
>>> >> >> >> >>> > >> > > *options);[/code]
>>> >> >> >> >>> > >> > >
>>> >> >> >> >>> > >> > > method.
>>> >> >> >> >>> > >> > >
>>> >> >> >> >>> > >> > > On Wed, Jan 19, 2011 at 11:41 AM, Szymon Kapeniak <
>>> >> >> >> >>> > >> > szymon.kapeniak at gmail.com
>>> >> >> >> >>> > >> > >> wrote:
>>> >> >> >> >>> > >> > >
>>> >> >> >> >>> > >> > >> greetings!
>>> >> >> >> >>> > >> > >>
>>> >> >> >> >>> > >> > >> what is a magic to let  my GU_Detail class to
>>> ::load()
>>> >> >> both
>>> >> >> >> files
>>> >> >> >> >>> from
>>> >> >> >> >>> > >> > >> disk as operator's streams with op:/path syntax? I
>>> can
>>> >> see
>>> >> >> >> load()
>>> >> >> >> >>> > >> > >> with an UT_IStream as argument, but I have no idea
>>> how
>>> >> to
>>> >> >> use
>>> >> >> >> it.
>>> >> >> >> >>> Does
>>> >> >> >> >>> > >> > >> any of UT_Stream versions handle both files and
>>> steams
>>> >> or
>>> >> >> the
>>> >> >> >> >>> logic is
>>> >> >> >> >>> > >> > >> on my side?
>>> >> >> >> >>> > >> > >>
>>> >> >> >> >>> > >> > >> thanks,
>>> >> >> >> >>> > >> > >> skk.
>>> >> >> >> >>> > >> > >> _______________________________________________
>>> >> >> >> >>> > >> > >> Sidefx-houdini-list mailing list
>>> >> >> >> >>> > >> > >> Sidefx-houdini-list at sidefx.com
>>> >> >> >> >>> > >> > >>
>>> >> >> >> >>>
>>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >> >>> > >> > >>
>>> >> >> >> >>> > >> > > _______________________________________________
>>> >> >> >> >>> > >> > > Sidefx-houdini-list mailing list
>>> >> >> >> >>> > >> > > Sidefx-houdini-list at sidefx.com
>>> >> >> >> >>> > >> > >
>>> >> >> >>
>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >> >>> > >> > >
>>> >> >> >> >>> > >> > _______________________________________________
>>> >> >> >> >>> > >> > Sidefx-houdini-list mailing list
>>> >> >> >> >>> > >> > Sidefx-houdini-list at sidefx.com
>>> >> >> >> >>> > >> >
>>> >> >> >>
>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >> >>> > >> >
>>> >> >> >> >>> > >> _______________________________________________
>>> >> >> >> >>> > >> Sidefx-houdini-list mailing list
>>> >> >> >> >>> > >> Sidefx-houdini-list at sidefx.com
>>> >> >> >> >>> > >>
>>> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >> >>> > >
>>> >> >> >> >>> > > --
>>> >> >> >> >>> > > Mark Elendt
>>> >> >> >> >>> > > _______________________________________________
>>> >> >> >> >>> > > Sidefx-houdini-list mailing list
>>> >> >> >> >>> > > Sidefx-houdini-list at sidefx.com
>>> >> >> >> >>> > >
>>> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >> >>> > >
>>> >> >> >> >>> > _______________________________________________
>>> >> >> >> >>> > Sidefx-houdini-list mailing list
>>> >> >> >> >>> > Sidefx-houdini-list at sidefx.com
>>> >> >> >> >>> >
>>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >> >>>
>>> >> >> >> >>> --
>>> >> >> >> >>> Mark Elendt
>>> >> >> >> >>> _______________________________________________
>>> >> >> >> >>> Sidefx-houdini-list mailing list
>>> >> >> >> >>> Sidefx-houdini-list at sidefx.com
>>> >> >> >> >>>
>>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >> >>>
>>> >> >> >> >> _______________________________________________
>>> >> >> >> >> Sidefx-houdini-list mailing list
>>> >> >> >> >> Sidefx-houdini-list at sidefx.com
>>> >> >> >> >>
>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >> >>
>>> >> >> >> >
>>> >> >> >> _______________________________________________
>>> >> >> >> Sidefx-houdini-list mailing list
>>> >> >> >> Sidefx-houdini-list at sidefx.com
>>> >> >> >>
>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >>
>>> >> >> > _______________________________________________
>>> >> >> > Sidefx-houdini-list mailing list
>>> >> >> > Sidefx-houdini-list at sidefx.com
>>> >> >> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >> >
>>> >> >> _______________________________________________
>>> >> >> Sidefx-houdini-list mailing list
>>> >> >> Sidefx-houdini-list at sidefx.com
>>> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >>
>>> >> > _______________________________________________
>>> >> > Sidefx-houdini-list mailing list
>>> >> > Sidefx-houdini-list at sidefx.com
>>> >> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >> >
>>> >> _______________________________________________
>>> >> Sidefx-houdini-list mailing list
>>> >> Sidefx-houdini-list at sidefx.com
>>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >>
>>> > _______________________________________________
>>> > Sidefx-houdini-list mailing list
>>> > Sidefx-houdini-list at sidefx.com
>>> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>> >
>>> _______________________________________________
>>> Sidefx-houdini-list mailing list
>>> Sidefx-houdini-list at sidefx.com
>>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>>
>>
>>
> _______________________________________________
> Sidefx-houdini-list mailing list
> Sidefx-houdini-list at sidefx.com
> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>



More information about the Sidefx-houdini-list mailing list