[Sidefx-houdini-list] HDK: op:/ magic

Olex P hoknamahn at gmail.com
Fri Jan 21 20:47:30 EST 2011


Well this bit of code

cout << "Process id = " << UT_Thread::getMyThreadId() << endl;

gave me different ids so I assume that VEX op was running on several
threads. I hope :)

On Sat, Jan 22, 2011 at 1:42 AM, Olex P <hoknamahn at gmail.com> wrote:

> Yes, test was heavy enough (1024x1024). So in theory it had to be running
> on 8 threads. But top gave me 15 to 90 percent of CPU used so I'm not sure
> was it really multithreaded. Pprobably it makes sense to do one more test
> that gives you the id of a thread it's running on.
> GDP object was allocated in Init(). Same with loading of geometry.
> I think you can also do first loading in Evaluate() in order to be able to
> use the file name passed to VEX op. Shouldn't be a problem. I'll check that
> tomorrow.
>
>
> On Sat, Jan 22, 2011 at 12:17 AM, Szymon Kapeniak <
> szymon.kapeniak at gmail.com> wrote:
>
>> That would be fantastic! But is your geometry enough heavy to start
>> multi-threading? Grid 100x100 should be fine, but anything smaller
>> doesn't use threads even if you set it on a vex node.
>>
>> Secondly, do you allocate new GU_Detail and load bgeo inside init()? I
>> would love to do that, but I don't know how to pass user arguments
>> there. It doesn't seem to take any, so all I have must take place
>> inside main function. I may be wrong also in this point though...
>>
>> I'll try your example tomorrow.
>>
>> Thank you Olex!
>>
>> 2011/1/22 Olex P <hoknamahn at gmail.com>:
>> > Hi Szymon,
>> >
>> > I've made a simple VEX dso which creates a static pointer to GU_Detail
>> >
>> > static GU_Detail* geo = NULL;
>> >
>> > allocates new object on Init()
>> >
>> > if(!geo)
>> > {
>> >  geo = new GU_Detail();
>> >  geo->load("/tmp/something.bgeo", 0);
>> > }
>> >
>> > takes an attribute value on Evaluate and returns it
>> >
>> > GEO_Point* pt = geo->points()[ptnum];
>> > GB_AttributeRef somethingIndex = geo->findPointAttrib("something",
>> > GB_ATTRIB_FLOAT);
>> > const float* something = pt->castAttribData<float>(somethingIndex);
>> > *result = *something;
>> >
>> > VEX op itself looks like:
>> >
>> > sop
>> > vextest(export float newsomething = 0)
>> > {
>> >    newsomething = myfun((float)ptnum);
>> > }
>> >
>> > So we create a geometry, apply an attribute on it, save somewhere. Then
>> > apply this VEX op to clean (without an attribute geo) and spreadsheet
>> show
>> > exactly the same values as where written into bgeo.
>> > It works fine.
>> > On Fri, Jan 21, 2011 at 6:30 PM, Szymon Kapeniak
>> > <szymon.kapeniak at gmail.com>wrote:
>> >
>> >> The problem is I don't have a chance to access any attribute, read,
>> >> write, what so ever. I declare gdp as static pointer and create new
>> >> GU_Detail inside init(), which meant to run once before threads start,
>> >> then I point threads to that gdp as advised in docs. It's enough to
>> >> crash Houdini in sop context.
>> >>
>> >> 2011/1/21 Olex P <hoknamahn at gmail.com>:
>> >> > Hi Szymon,
>> >> >
>> >> > VEX itself has nothing to do with multithreading. But VEX engine
>> >> (standard
>> >> > contexts or your own CVEX one) does. So in the case you mentioned
>> (VEX
>> >> SOP
>> >> > that is reading a point cloud and assigning a filtered value to a
>> point)
>> >> it
>> >> > works as (simplified):
>> >> > 1. VEX engine (SOP context) creates a flat array and fills it in with
>> the
>> >> > attributes from geometry
>> >> > 2. VEX engine spawns several threads and runs one instance of VEX
>> >> procedure
>> >> > in each thread
>> >> > 3. Each instance of VEX procedure (single threaded) filters a value
>> from
>> >> > point cloud and writes into it's chunk of flat array
>> >> > 4. VEX engine writes all the values back from flat array into a GDP
>> using
>> >> > one thread
>> >> > Honestly I haven't tried multithreading with GDP but I see no
>> problems
>> >> with
>> >> > it as long as you are not modifying the data. And normally VEX
>> function
>> >> > itself doesn't modify a GDP (but VEX engine does).
>> >> > Well that's how I understand it and I can be wrong :)
>> >> > There are some bits about thread safety in HDK
>> >> >
>> >>
>> http://www.sidefx.com/docs/hdk11.0/hdk_changes_11_0.html#HDKC11_0_ThreadSafety
>> >> > I probably have to do some tests too :)
>> >> >
>> >> > On Fri, Jan 21, 2011 at 5:13 PM, Szymon Kapeniak
>> >> > <szymon.kapeniak at gmail.com>wrote:
>> >> >
>> >> >> Hey Olex,
>> >> >> I'm writing custom vex, and I'm sure vex has some standard methods
>> to
>> >> >> deal with this. I'm only interested in read access from several
>> >> >> threads which I don't have any control on, as they are spawned by
>> VEX
>> >> >> SOP.
>> >> >>
>> >> >> I started to play with it by checking minimal scenarios and it seems
>> >> >> it's enough to declare one static GU_Detail instance and assign
>> >> >> pointers to it from a couple of threads to crash Houdini even
>> without
>> >> >> loading any geometry.
>> >> >> Also by checking threads ids I realized, that mantra can run the
>> same
>> >> >> code multi-threaded because vex in surface context call init()
>> >> >> (standard procedure) per thread, unlike in sop context, where it
>> seems
>> >> >> init() is called only once like in fact it's described in docs.
>> >> >>
>> >> >> Surly there are thread-save vex functions (reading gdp) which
>> operate
>> >> >> in sop context, like pcopen().  I'm guessing the trick is not to use
>> >> >> pure gdp, but some  thread-safe memory schema. Lack of documentation
>> >> >> doesn't play well with my ignorance...
>> >> >>
>> >> >> skk.
>> >> >>
>> >> >> btw  I'm referring to this talk:
>> >> >>
>> >> >>
>> >>
>> http://forums.odforce.net/index.php?/topic/12434-how-to-replicate-this-ice-network-in-vops/page__st__12
>> >> >>
>> >> >> 2011/1/21 Olex P <hoknamahn at gmail.com>:
>> >> >> > Hey Szymon,
>> >> >> >
>> >> >> > You can read in multiple threads but multithread writing as far as
>> I
>> >> know
>> >> >> is
>> >> >> > limited to volume primitives (each thread can write into it's own
>> >> tile)
>> >> >> as
>> >> >> > far as I know. You could create own GU/GEO_Detail object for each
>> >> thread,
>> >> >> > write into it and then merge all the pieces of puzzle into one
>> object
>> >> >> (merge
>> >> >> > will single thread).
>> >> >> > But from your words I'm not sure what kind of parallelism you are
>> >> looking
>> >> >> > for...
>> >> >> > Let's say sampling from volume in VEX. 1. You are running the same
>> >> single
>> >> >> > thread code on different chunks of data using multiple threads. 2.
>> But
>> >> >> this
>> >> >> > also could be done in a different way: vex function internally
>> spawns
>> >> >> > several threads which modify the data somehow. You probably
>> talking
>> >> about
>> >> >> 2
>> >> >> > (as you mentioned multithread access to GU_Detail). I'm not sure
>> but
>> >> >> maybe
>> >> >> > it makes sense to use 1 (i.e. CVEX). Sure it will work only for
>> data
>> >> >> > parallel algorithms.
>> >> >> >
>> >> >> > On Thu, Jan 20, 2011 at 10:20 PM, Szymon Kapeniak <
>> >> >> szymon.kapeniak at gmail.com
>> >> >> >> wrote:
>> >> >> >
>> >> >> >> ...sorry for spamming, but I've just realized that the same dso
>> works
>> >> >> >> in mantra, but not in (threading) sop context. Kind of strange.
>> >> >> >>
>> >> >> >> 2011/1/20 Szymon Kapeniak <szymon.kapeniak at gmail.com>:
>> >> >> >> > Thanks for a response, I didn't know I have to deal with it
>> >> manually.
>> >> >> >> >
>> >> >> >> > Second problem is threading in vex sop. After playing with my
>> >> function
>> >> >> >> > a little, I'm getting a feeling my approach was too naive, so
>> to
>> >> >> >> > speak.
>> >> >> >> >
>> >> >> >> > Is multi-threaded access to a static GU_Detail * possible at
>> all,
>> >> or
>> >> >> >> > do I have to use some caches like UT_ThreadSafeCache or
>> anything
>> >> >> >> > similar? What is a general way of doing this in HDK?
>> >> >> >> >
>> >> >> >> > thanks,
>> >> >> >> > skk.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > 2011/1/19 Olex P <hoknamahn at gmail.com>:
>> >> >> >> >> It kind of does writing an op-referenced texture into an IFD
>> :)
>> >> >> >> >>
>> >> >> >> >> On Wed, Jan 19, 2011 at 1:48 PM, Mark Elendt <mark at sidefx.com
>> >
>> >> >> wrote:
>> >> >> >> >>
>> >> >> >> >>> For mantra, even Houdini's VEX doesn't support the op: syntax
>> :)
>> >> >> >> >>>
>> >> >> >> >>> For VEX functions in OPs, you should be able to access the
>> SOP by
>> >> >> >> >>> grabbing the OP director and finding the SOP.
>> >> >> >> >>>
>> >> >> >> >>> On Wednesday Jan 19 at 13:39, Szymon Kapeniak wrote:
>> >> >> >> >>> > thanks Mark,
>> >> >> >> >>> > but how about custom vex dso? My func() opens file from
>> disk,
>> >> but
>> >> >> >> >>> > doesn't handle op: syntax. Is it doable in custom code?
>> >> >> >> >>> >
>> >> >> >> >>> > thanks,
>> >> >> >> >>> > skk.
>> >> >> >> >>> >
>> >> >> >> >>> > 2011/1/19 Mark Elendt <mark at sidefx.com>:
>> >> >> >> >>> > > Unfortunately, at the current time op: is handled as a
>> >> special
>> >> >> >> case,
>> >> >> >> >>> > > so it's not part of the generalized loader.
>> >> >> >> >>> > >
>> >> >> >> >>> > > You'll have to reference the geometry directly (i.e. cook
>> the
>> >> >> SOP).
>> >> >> >> >>> > > If you need your own copy, you'll have to merge it.
>> >> >> >> >>> > >
>> >> >> >> >>> > > On the other hand, opdef: is handled properly.
>> >> >> >> >>> > >
>> >> >> >> >>> > > On Wednesday Jan 19 at 12:11, Olex P wrote:
>> >> >> >> >>> > >> Maybe you right Szymon. Most of the usage of op: syntax
>> for
>> >> me
>> >> >> was
>> >> >> >> >>> sampling
>> >> >> >> >>> > >> from point clouds/volumes/textures and I have never
>> tries to
>> >> >> load
>> >> >> >> >>> geometry
>> >> >> >> >>> > >> using op: syntax. File SOP doesn't work with it and I
>> think
>> >> >> it's
>> >> >> >> >>> because of
>> >> >> >> >>> > >> the fact that load() expects a proper file format. Would
>> be
>> >> >> great
>> >> >> >> to
>> >> >> >> >>> hear
>> >> >> >> >>> > >> about this from SESI.
>> >> >> >> >>> > >>
>> >> >> >> >>> > >> On Wed, Jan 19, 2011 at 11:58 AM, Szymon Kapeniak <
>> >> >> >> >>> szymon.kapeniak at gmail.com
>> >> >> >> >>> > >> > wrote:
>> >> >> >> >>> > >>
>> >> >> >> >>> > >> > hi Olex,
>> >> >> >> >>> > >> > thanks for fast response. Well, I must be doing
>> something
>> >> >> >> stupid,
>> >> >> >> >>> > >> > because it doesn't.
>> >> >> >> >>> > >> >
>> >> >> >> >>> > >> > GU_Detail gdp;
>> >> >> >> >>> > >> > const char *file   = (const char *) argv[1];
>> >> >> >> >>> > >> > if (gdp.load(file, 0) < 0) return ;
>> >> >> >> >>> > >> > ...
>> >> >> >> >>> > >> >
>> >> >> >> >>> > >> > while works fine with disk paths, doesn't work with
>> >> op:/'ed
>> >> >> one.
>> >> >> >> >>> > >> >
>> >> >> >> >>> > >> >
>> >> >> >> >>> > >> > 2011/1/19 Olex P <hoknamahn at gmail.com>:
>> >> >> >> >>> > >> > > Hi Szymon,
>> >> >> >> >>> > >> > >
>> >> >> >> >>> > >> > > I don't think that you need any magic. op: syntax
>> has to
>> >> be
>> >> >> >> >>> handled
>> >> >> >> >>> > >> > > automagically when you using
>> >> >> >> >>> > >> > >
>> >> >> >> >>> > >> > > [code]virtual int          load(const char *, const
>> >> >> UT_Options
>> >> >> >> >>> > >> > > *options);[/code]
>> >> >> >> >>> > >> > >
>> >> >> >> >>> > >> > > method.
>> >> >> >> >>> > >> > >
>> >> >> >> >>> > >> > > On Wed, Jan 19, 2011 at 11:41 AM, Szymon Kapeniak <
>> >> >> >> >>> > >> > szymon.kapeniak at gmail.com
>> >> >> >> >>> > >> > >> wrote:
>> >> >> >> >>> > >> > >
>> >> >> >> >>> > >> > >> greetings!
>> >> >> >> >>> > >> > >>
>> >> >> >> >>> > >> > >> what is a magic to let  my GU_Detail class to
>> ::load()
>> >> >> both
>> >> >> >> files
>> >> >> >> >>> from
>> >> >> >> >>> > >> > >> disk as operator's streams with op:/path syntax? I
>> can
>> >> see
>> >> >> >> load()
>> >> >> >> >>> > >> > >> with an UT_IStream as argument, but I have no idea
>> how
>> >> to
>> >> >> use
>> >> >> >> it.
>> >> >> >> >>> Does
>> >> >> >> >>> > >> > >> any of UT_Stream versions handle both files and
>> steams
>> >> or
>> >> >> the
>> >> >> >> >>> logic is
>> >> >> >> >>> > >> > >> on my side?
>> >> >> >> >>> > >> > >>
>> >> >> >> >>> > >> > >> thanks,
>> >> >> >> >>> > >> > >> skk.
>> >> >> >> >>> > >> > >> _______________________________________________
>> >> >> >> >>> > >> > >> Sidefx-houdini-list mailing list
>> >> >> >> >>> > >> > >> Sidefx-houdini-list at sidefx.com
>> >> >> >> >>> > >> > >>
>> >> >> >> >>>
>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >> >>> > >> > >>
>> >> >> >> >>> > >> > > _______________________________________________
>> >> >> >> >>> > >> > > Sidefx-houdini-list mailing list
>> >> >> >> >>> > >> > > Sidefx-houdini-list at sidefx.com
>> >> >> >> >>> > >> > >
>> >> >> >>
>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >> >>> > >> > >
>> >> >> >> >>> > >> > _______________________________________________
>> >> >> >> >>> > >> > Sidefx-houdini-list mailing list
>> >> >> >> >>> > >> > Sidefx-houdini-list at sidefx.com
>> >> >> >> >>> > >> >
>> >> >> >>
>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >> >>> > >> >
>> >> >> >> >>> > >> _______________________________________________
>> >> >> >> >>> > >> Sidefx-houdini-list mailing list
>> >> >> >> >>> > >> Sidefx-houdini-list at sidefx.com
>> >> >> >> >>> > >>
>> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >> >>> > >
>> >> >> >> >>> > > --
>> >> >> >> >>> > > Mark Elendt
>> >> >> >> >>> > > _______________________________________________
>> >> >> >> >>> > > Sidefx-houdini-list mailing list
>> >> >> >> >>> > > Sidefx-houdini-list at sidefx.com
>> >> >> >> >>> > >
>> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >> >>> > >
>> >> >> >> >>> > _______________________________________________
>> >> >> >> >>> > Sidefx-houdini-list mailing list
>> >> >> >> >>> > Sidefx-houdini-list at sidefx.com
>> >> >> >> >>> >
>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >> >>>
>> >> >> >> >>> --
>> >> >> >> >>> Mark Elendt
>> >> >> >> >>> _______________________________________________
>> >> >> >> >>> Sidefx-houdini-list mailing list
>> >> >> >> >>> Sidefx-houdini-list at sidefx.com
>> >> >> >> >>>
>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >> >>>
>> >> >> >> >> _______________________________________________
>> >> >> >> >> Sidefx-houdini-list mailing list
>> >> >> >> >> Sidefx-houdini-list at sidefx.com
>> >> >> >> >>
>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> _______________________________________________
>> >> >> >> Sidefx-houdini-list mailing list
>> >> >> >> Sidefx-houdini-list at sidefx.com
>> >> >> >>
>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >>
>> >> >> > _______________________________________________
>> >> >> > Sidefx-houdini-list mailing list
>> >> >> > Sidefx-houdini-list at sidefx.com
>> >> >> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >> >
>> >> >> _______________________________________________
>> >> >> Sidefx-houdini-list mailing list
>> >> >> Sidefx-houdini-list at sidefx.com
>> >> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >>
>> >> > _______________________________________________
>> >> > Sidefx-houdini-list mailing list
>> >> > Sidefx-houdini-list at sidefx.com
>> >> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >> >
>> >> _______________________________________________
>> >> Sidefx-houdini-list mailing list
>> >> Sidefx-houdini-list at sidefx.com
>> >> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >>
>> > _______________________________________________
>> > Sidefx-houdini-list mailing list
>> > Sidefx-houdini-list at sidefx.com
>> > https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>> >
>> _______________________________________________
>> Sidefx-houdini-list mailing list
>> Sidefx-houdini-list at sidefx.com
>> https://lists.sidefx.com:443/mailman/listinfo/sidefx-houdini-list
>>
>
>



More information about the Sidefx-houdini-list mailing list