[DONE] Issue:Perfomance:Import & analysis:Elephant & PostgreSQL load CPU up to 100%

    • stanisluv
      stanisluv
      Moderator
      Moderator
      Joined: 18.06.2008 Posts: 45,066
      Hi all.

      1. Download the following archive: http://www.sendspace.com/file/h6pke3
      2. Extract this archive to a single folder.
      3. Import this folder into Elephant.
      4. Open task manager.

      You should detect following situation:



      Elephant & PostgreSQL load CPU up to 100% when import & analyze data from every room. For example: create new DB & restart Elephant. You`l see the same situation.


      More than that you could detect that Elepahnt analyze this hands very slow: about 100 hands per 2.5 mins. This is with Party Poker Hand Hstory only.
  • 16 replies
    • Larres
      Larres
      Bronze
      Joined: 07.09.2009 Posts: 2,402
      Hi there,

      we'll have a try if it's reproducable on our side as well.

      Thx,
      L
    • Larres
      Larres
      Bronze
      Joined: 07.09.2009 Posts: 2,402
      Hi there,

      I just put the data in and the import runs very smoothly using the latest Elephant builds:

      294 imported files
      non not imported files
      298.054 imported hands
      108 not imported files
      8 minutes and 7 seconds import duration

      As you can see our new import is fast as lightning.

      The legacy analysis on the other hand is still working - as expected.
      You may already know that we're currently working on a complete new analysis engine, the prerequisite for the new tourney HUD. And we will enhance it's performance as much as we already did for the import engine.

      But at the moment the legacy analysis engine is a known bottle neck and the more data the legacy analysis engine gets the slower it runs.
      Analysing smaller packages with a maximum of ~50.000 hands will run much better for the legacy.

      Cheers,
      L

      PS: The legacy analysis of the 300.000 hands is nearly working an hour now and still running
    • stanisluv
      stanisluv
      Moderator
      Moderator
      Joined: 18.06.2008 Posts: 45,066
      Strange. I test in on too PCs with dual-core CPU & with Quad CPU. The both PCs shows the similar results for legacy analysis engine. ?( ?(

      What about CPU usage? As I see you use Quad CPU . Elephant & PostgreSQL should load CPU up to 90% during import period.
    • Larres
      Larres
      Bronze
      Joined: 07.09.2009 Posts: 2,402
      Hi there,

      the usage stays is between 55 and 65%.
      But there is indeed an errorneous file reported by the analysis:

      ERROR: Exception caught in Void PS.Database.nHibernate.Repositories.HandRepository.Update(NHibernate.ISession, System.Collections.Generic.IEnumerable`1[PS.Poker.IHand])
      -------------------- PS.Common.Database.DatabaseException --------------------
      - Message: identifier of an instance of PS.Poker.Hand.Hand was altered from 9816976768_0_1_3_0 to 9816976768_0_1_0_0

      Can you arrange and send us the original hand history holding the game with the above mentioned ID '9816976768'.

      Thx,
      L

      PS: Analysis ist still running...
    • stanisluv
      stanisluv
      Moderator
      Moderator
      Joined: 18.06.2008 Posts: 45,066
      Unfortunally I cann`t get original HH but I request this hand via e-mail. the received hand has the same structure as in the file 218SH2010.11.0713.txt. Do you need claimed hand?

      UPD: Screenshot from "Quad PC"

    • Larres
      Larres
      Bronze
      Joined: 07.09.2009 Posts: 2,402
      Hi stanisluv,

      the cpu usage is related to the analysis, as there are issues analysing the hands.
      In order to invesigate this further we need the originals.

      And Elephant v0.97 is released, I suggest to switch to the latest build.

      Cheers,
      L
    • stanisluv
      stanisluv
      Moderator
      Moderator
      Joined: 18.06.2008 Posts: 45,066
      Hi Larres,

      Ok, I agree with you about PartyPoker but what about PokerStars? I play on PokerStars & have the same issue.

      Today I have tried to import about 73k hands from 600 files & got following results:

      Import duration: 8min
      Analysis duration: 2h 20min. ;( ;(

      The CPU usage due to first 8 mins(import+analysis): 90-100%

      What do you think about this?
    • Larres
      Larres
      Bronze
      Joined: 07.09.2009 Posts: 2,402
      Hi stanisluv,

      don't get me wrong. The slow analysis is a big issue here and we're indeed about to rewrite the whole analysis engine as a prerequisite for the new tourney HUD. And as soon as the new analysis engine is ready to use this bottle neck will be gone.

      Cheers,
      L
    • stanisluv
      stanisluv
      Moderator
      Moderator
      Joined: 18.06.2008 Posts: 45,066
      Forgot to notify: somebody up Elephant perfomance in v 0.98 & now I I have following results:

      Hands: 76k in 635 files
      Import duration: 8min
      Analysis duration: 1h 50min.

      Nice work guys!! :s_thumbsup: :s_thumbsup:
      :s_thumbsup:
    • MichaelWagner42
      MichaelWagner42
      Bronze
      Joined: 12.09.2009 Posts: 1,552
      Hi stanisluv,

      We will very likely publis the 0.99 in around 2 weeks time. Can you repeat that speed test with the 0.99? We have tweaked something on the legacy import part and this shall be a little faster now.

      Ciao
      Michael
    • stanisluv
      stanisluv
      Moderator
      Moderator
      Joined: 18.06.2008 Posts: 45,066
      Hi Michael,

      Yes, of course. Thx for info & timeline.

      Stanislav.
    • Larres
      Larres
      Bronze
      Joined: 07.09.2009 Posts: 2,402
      Hi stanisluv,

      the v0.99 is already available through our blog and I'd like you to check the improved version for new results

      Cheers,
      L
    • stanisluv
      stanisluv
      Moderator
      Moderator
      Joined: 18.06.2008 Posts: 45,066
      Already checked & reported. ;)
    • Larres
      Larres
      Bronze
      Joined: 07.09.2009 Posts: 2,402
      So I assume this one is done?!
      Anyway the next version comes with a fix for the memory constumption.

      L
    • stanisluv
      stanisluv
      Moderator
      Moderator
      Joined: 18.06.2008 Posts: 45,066
      Yes, you can. :)
    • Larres
      Larres
      Bronze
      Joined: 07.09.2009 Posts: 2,402
      okey dokey