Cache-EX Loopback, or no loopback.. ?

[m404]

Can somebody else do this simple test mentioned earlier ?

it's simple, really :

de-duping cache entries is done by use of the nodeID.
because the nodeID unfortunately doesn't always turn out to be the same for reader/account (reasons still being investigated), the de-duping done by oscam itself fails.
using intelligent groups makes sure this does not happen.
the internal system does not require "same group" to do its internal de-duping, btw ... it will base its decision on the nodeID, whether the groups are correct or not ...

[CapNCooK]

I identify two very different things in this discussion;

1. The de-duping. (e.g. rejecting incoming cache that already exists in the pool)
2. Loopback (e.g. (preventing to) return cache to the same peer it originally came from)

----

Let's first talk about de-duplication, and its relation to groups.;

de-duping cache entries is done by use of the nodeID.
because the nodeID unfortunately doesn't always turn out to be the same for reader/account (reasons still being investigated), the de-duping done by oscam itself fails.

How ? In my understanding, de-duping is based on Checksum.. and mayb CSP_Hash.. NodeID is beeing used to prevent loopback.

----

Using intelligent groups makes sure this does not happen.

Can you motivate that statement a little more ? I don't understand.. especially when reading your next statement;

the internal system does not require "same group" to do its internal de-duping, btw ... it will base its decision on the nodeID, whether the groups are correct or not ...

That's quite the opposite of what you were saying in previous statement.

------

See the results of this test;

• I have connected my cache-ex server twice to to my sharing server.
  So.. on my sharing server, i have two exactly the same incoming cache streams.
• I have assigned one receiving-user in group 50, and the other receiving-user in group 51.

Now see the cache log, monitoring a specific SID:

2013/04/02 15:39:31 2542C20 c got pushed ECM 0100:00006A:07D5 -- 3171E5EB5CE0616E75EDF4BB506F53C2_EC77C36 F from X9-CACHE-EXCHANGE2
2013/04/02 15:39:32 2530320 c got pushed ECM 0100:00006A:07D5 -- 3171E5EB5CE0616E75EDF4BB506F53C2_EC77C36 F from X9-CACHE-EXCHANGE
2013/04/02 15:39:32 2542C20 c got pushed ECM 0100:00006A:07D5 -- 6C6C155346D928FA664FF19DB8E3E87A_2754951 B from X9-CACHE-EXCHANGE2
2013/04/02 15:39:33 2530320 c got pushed ECM 0100:00006A:07D5 -- 6C6C155346D928FA664FF19DB8E3E87A_2754951 B from X9-CACHE-EXCHANGE
2013/04/02 15:39:37 2542C20 c got pushed ECM 0100:00006A:07D5 -- 68E0FFBC8B62C92AF0028BB96674838F_2ECB8BD 5 from X9-CACHE-EXCHANGE2
2013/04/02 15:39:38 2530320 c got pushed ECM 0100:00006A:07D5 -- 68E0FFBC8B62C92AF0028BB96674838F_2ECB8BD 5 from X9-CACHE-EXCHANGE
2013/04/02 15:39:45 2530320 c got pushed ECM 0100:00006A:07D5 -- CF7A6A5BE9CD56950A0AB5823808B6B9_2952686 5 from X9-CACHE-EXCHANGE
2013/04/02 15:39:45 2542C20 c got pushed ECM 0100:00006A:07D5 -- CF7A6A5BE9CD56950A0AB5823808B6B9_2952686 5 from X9-CACHE-EXCHANGE2
2013/04/02 15:39:49 2530320 c got pushed ECM 0100:00006A:07D5 -- 7A411396A138904C6F4763180E6C8BE9_D9FA8A1 E from X9-CACHE-EXCHANGE
2013/04/02 15:39:50 2542C20 c got pushed ECM 0100:00006A:07D5 -- 7A411396A138904C6F4763180E6C8BE9_D9FA8A1 E from X9-CACHE-EXCHANGE2

As you can see.. Each CW is accepted twice.. One accepted for group 50, and one accepted for group 51.
Result: Duplicate cached cw's in your pool / Duplicate outgoing traffic / Duplicate load.

--

Now we do the test again;

• assigned both receiving-users to group 50.

See the log:

2013/04/02 15:48:54 2530320 c got pushed ECM 0100:00006A:07D5 -- D632AEABFF16C8FCB11111E4BE58CEE6_1F1C2DC 6 from X9-CACHE-EXCHANGE (off)
2013/04/02 15:48:55 2542C20 c ignored duplicate pushed ECM 0100:00006A:07D5 -- D632AEABFF16C8FCB11111E4BE58CEE6_1F1C2DC 6 from X9-CACHE-EXCHANGE2
2013/04/02 15:48:56 2542C20 c got pushed ECM 0100:00006A:07D5 -- A215D6F0D0DB55BB79AAAA8CE8510A53_09A7A71 4 from X9-CACHE-EXCHANGE2 (off)
2013/04/02 15:48:56 2530320 c ignored duplicate pushed ECM 0100:00006A:07D5 -- A215D6F0D0DB55BB79AAAA8CE8510A53_09A7A71 4 from X9-CACHE-EXCHANGE
2013/04/02 15:48:57 2530320 c got pushed ECM 0100:00006A:07D5 -- B9CA4E998B2312D6099FF157B2B4C6E2_C284AA5 7 from X9-CACHE-EXCHANGE (off)
2013/04/02 15:48:58 2542C20 c ignored duplicate pushed ECM 0100:00006A:07D5 -- B9CA4E998B2312D6099FF157B2B4C6E2_C284AA5 7 from X9-CACHE-EXCHANGE2
2013/04/02 15:49:02 2530320 c got pushed ECM 0100:00006A:07D5 -- 296BEA11366C9509455F473AC6F49796_4633677 B from X9-CACHE-EXCHANGE (off)
2013/04/02 15:49:02 2542C20 c ignored duplicate pushed ECM 0100:00006A:07D5 -- 296BEA11366C9509455F473AC6F49796_4633677 B from X9-CACHE-EXCHANGE2
2013/04/02 15:49:04 2530320 c got pushed ECM 0100:00006A:07D5 -- C1C226DEF761E8616A9E6B97214B0191_647672A 7 from X9-CACHE-EXCHANGE (off)
2013/04/02 15:49:04 2542C20 c ignored duplicate pushed ECM 0100:00006A:07D5 -- C1C226DEF761E8616A9E6B97214B0191_647672A 7 from X9-CACHE-EXCHANGE2
2013/04/02 15:49:08 2530320 c got pushed ECM 0100:00006A:07D5 -- E9A549398B9AE8C7DC1EA2AE07A5451F_4E57866 4 from X9-CACHE-EXCHANGE (off)
2013/04/02 15:49:08 2542C20 c ignored duplicate pushed ECM 0100:00006A:07D5 -- E9A549398B9AE8C7DC1EA2AE07A5451F_4E57866 4 from X9-CACHE-EXCHANGE2
2013/04/02 15:49:11 2542C20 c got pushed ECM 0100:00006A:07D5 -- 42638885873AAD58F75732668C802F41_1863741 C from X9-CACHE-EXCHANGE2 (off)
2013/04/02 15:49:11 2530320 c ignored duplicate pushed ECM 0100:00006A:07D5 -- 42638885873AAD58F75732668C802F41_1863741 C from X9-CACHE-EXCHANGE


So.. You can all say what you want.. But according to my logs and understanding.. groups do matter.

I will create another post in this thread regarding (2) - Loopback, since this was all and only about de-duplication.

[m404]

I identify two very different things in this discussion;

1. The de-duping. (e.g. rejecting incoming cache that already exists in the pool)
2. Loopback (e.g. (preventing to) return cache to the same peer it originally came from)

these two are the same thing, the only difference is that the second case is more defined, more specific (apart from being a dupe entry, it's also coming from the same peer it was sent to in the first place) ... but a loopback cache has to be ultimately also a dupe, hence why the de-duping already takes care of it :)

I can agree with this, and this bug could be there. (is this discussed somewhere on SB?)

think it was mentioned somewhere, but unfortunately it was sort of "forgotten" about, due to not being able to reproduce it.
as you know, SB doesn't officially support this big-scale sharing ... not for normal cardsharing, not for cacheex changing ... so it got muffled somewhere along the lines when some mod "reminded" ppl about the rules :S

- Can you motivate that statement a little more ? I don't understand.. especially when reading your next statement;
- That's quite the opposite of what you were saying in previous statement.

it's not, you must've misunderstood me :)

nodeID takes care of the de-duping. the nodeID is part of the md5 hash being calculated, hence, assuming that the nodeID for reader and user are identical, it will never happen that it is sent back ... not when it's the same peer it was received from, nor if the peer is available somewhere else in the circle ...
unfortunately, nodeID being unique is an assumption that is failing (as we already said, and know) ... therefore, it still happens.
if you create the correct groups, then you have a more manual control about it ...

reguarding your test :

So.. You can all say what you want.. But according to my logs and understanding.. groups do matter.

i agree, groups DO matter :)
what they don't matter for, or shouldn't matter for, is de-duping (why would the CWs you are showing in your log not be seen as dupes? they come from the same nodeID, they are the same CW, and the md5 hash should be identical, since the group is not part of it ... why your oscam considers them not to be dupes, is beyond my knowledge).

i think we should report your findings to SB, because they're definitely not according to how dupe-identification is supposed to work ...
(just so we're clear : i agree, your findings would suggest that reader and account need to be in the same group ... but the SB wiki itself will tell you it's not supposed to be like that :D so i believe something isn't working as it's meant to, in oscam ... what a surprise xD )

[CapNCooK]

Now about the other thing..

2 - Loopback (e.g.: preventing cache to be send-back to its origin)

--

Another simple test mentioned earlier, now with logs;

Setup:

- 1 Cache-EX server
- 1 Receiver with oscam

both connected to each other (bi-directional), and no other user/reader in receiver-oscam.

Now see this log from receiver-oscam:

2013/04/02 17:13:45 2714BC0 c got pushed ECM 0100:00006A:07E9 -- A0A5D223D75A1DC810D8C530A09B0AB3_38D3C69 F from X9-CACHE-EXCHANGE (off)
2013/04/02 17:13:45 2545F30 p cacheex: check node 16064601546241928702X == 16064601546241928702X ?
2013/04/02 17:13:45 2545F30 p cacheex: node 16064601546241928702X found in list => skip push!
2013/04/02 17:13:45 2714BC0 c got pushed ECM 0963:000000:C7A8 -- EB65019F9C5CB3000FACBB747F817EA4_5940420 F from X9-CACHE-EXCHANGE (off)
2013/04/02 17:13:45 2545F30 p cacheex: check node 16064601546241928702X == 16064601546241928702X ?
2013/04/02 17:13:45 2545F30 p cacheex: node 16064601546241928702X found in list => skip push!



As you can see.. None is beeing send back. So there is no loopback.

--

Now i add a hardware reader with card to the receiver-oscam, and a user watching one of the channels on card:

As you can see from this receiver-oscam log, the CW's from card are nicely beeing send back to the cache server, while the mass is still beeing rejected:

2013/04/02 17:07:26 270D000 c USER1 (0963:000000:D3C9 -- E33FEDCAB588914D1CF6C912D69F71CB_7734F42 C): found (269 ms) by card1 (P/1/1/5) - Nollywood Channel (real 19 ms)
2013/04/02 17:07:26 2545F30 p pushed ECM 0963:000000:D3C9 -- E33FEDCAB588914D1CF6C912D69F71CB_7734F42 C to X9-CACHE-RETURN res 69 stats -1


This is also beeing proved from CACHE-EX page in webif:

211146 CW Received from CACHE-EX server
92 local decodes are beeing send back.






So.. If anyone still want to state otherwise.. please include some logs with proof to let me fully understand what you're saying.

Thx!

[m404]

i dont understand what this last post is supposed to be showing?
of course the locally (receiver) created CW is being sent to the cache server ... why shouldnt it be?
and ofc no loopback is happening ... see your log, it clearly says it checks the node, sees they're identical, and therefore skips the push?

again ... what's this supposed to say? :)

[CapNCooK]

these two are the same thing, the only difference is that the second case is more defined, more specific (apart from being a dupe entry, it's also coming from the same peer it was sent to in the first place) ... but a loopback cache has to be ultimately also a dupe, hence why the de-duping already takes care of it :)

I do not agree.

- De-Duping takes place at first/earliest CW entrance in oscam
- Preventing loopback takes place after that the CW is accepted (and already beeing marked als non-duplicate.)

nodeID takes care of the de-duping.
/ .. /
Not when it's the same peer it was received from, nor if the peer is available somewhere else in the circle ...

Replace "de-duping" into "loopback-prevention", and i agree with this statement

unfortunately, nodeID being unique is an assumption that is failing (as we already said, and know) ... therefore, it still happens.
if you create the correct groups, then you have a more manual control about it ...

Aside from the bug you mentioned, which could be true, how would groups give more control.. Its the opposite.. When using multiple groups, you loose control! (of de-duplication)
Loopback-prevention still works, oscam never sends back cache to its origin NodeID like you said and agreed already.

reguarding your test :

i agree, groups DO matter :)
what they don't matter for, or shouldn't matter for, is de-duping (why would the CWs you are showing in your log not be seen as dupes?
they come from the same nodeID, they are the same CW, and the md5 hash should be identical, since the group is not part of it ...
(why your oscam considers them not to be dupes, is beyond my knowledge).

Now that's a good question, and also the culprit of this all. It SHOULD be seen as duplicates in my opinion, regardless of ANY groups.

But ever since i started to use CE, technically, cache is beeing considered UNIQUE PER GROUP.

So the ONLY way to prevent duplicates.. is to use.. ONE group. :)

PS: the wiki lacks any descent documentation about CE, and should be considered useless in these "under-the-hood" discussions.

[CapNCooK]

i dont understand what this last post is supposed to be showing?
again ... what's this supposed to say? :)

You are actually the only one beeing convinced / agreeing that loopback-prevention works.
So like said.. as i see the two as totally different things, its to prove the other misbelievers in this thread :)