True winrate - Sample size

    • cindy1985
      cindy1985
      Bronze
      Joined: 13.11.2007 Posts: 358
      How many hands do I need do know my winrate?
      Am I a winning player?

      These are some basic/usual/recurrent questions for someone thinking of turning pro, or some beginner worrying about his performance...

      I have been thinking on this subject lately, and since I can't reach my favourite poker room because of an earthquake near Taiwan, I am laying down my thoughts. Hoping it might interest someone although it is certainly not very well exposed.

      You don't need to understand the statistical concept behind to understand the conclusions... if you don't bother about the details, go to the conclusions directly :) Also figures are coming from my knowledge of usual winrate/standard deviation that I observed on some players in FR NL.



      First: why is it so hard to know our very own winrate? Because poker winnings are subjected to huge variance. On a 100 000 hands sample, if your winrate is somewhere close to 5BB/100 hands, the standard deviation is more close to 80BB... so if we use classical confidence interval, on a given sample size n, and assuming that the winnings are normally distributed (more on this later), our "true" winrate (in BB/100 hands) lies - with 95% confidence - somewhere in between:

      100*(0.05 - 1.96*8/sqrt(n)) < winrate < 100*(0.05 + 1.96*8/sqrt(n))

      Replacing n, we get the following interval:

      0.05 BB/100 mains < winrate < 10 BB/100 mains

      That is to say that the only 95% confidence info we get from a 100 000 hand sample (with a 5BB/100 hands observed on this sample), is that we are a winning (or even a breakeven) player.
      For some player, with a lower variance style, the situation is better, but still nothig spectacular. Say we divide the variance by two for a very solid FR player, we still have a wide interval:

      2.5 < winrate < 7.5


      Second: About the normality assumption.
      It is absurdely false, at least in the few samples I have observed. However, the beauty of the statistics is that if we sum 100 consecutive hands for example, then it tends to become normal (the famous CLM).
      I have made this transformation on my samples. Confidence intervals are similar to those explained above.


      Third: How many hands do we need to detect a small difference in our winrate with a given frequency?
      Basically, if we apply a statistical test, we want to guarantee we can detect a 1BB difference on our winrate with a given frequency (say 80%). This question is answered by the power of the test.
      Since we use approximately normal variables now (the sum of 100 consecutive hands), we want to use for example the t test (to test if our winrate is greater than a given 5BB/100 hands for example). It yields the following conclusions.


      Conclusions:

      Basically computing the power of the t test, we obtain the following info:

      - to detect a 1BB difference on our winrate (still with the 80BB standard deviation), we need 100 000 cluster of 100 variables, i.e. 10 million hands...
      - to detect a 2BB difference: 2 500 000 hands
      - ...

      Similarly, we can get the following info: for a given sample size, what difference can we detect
      - 100 000 hands => 10BB!!!
      - 1 000 000 hands => 3BB
      - 2 000 000 hands => 2.2BB

      Hope this can give you a "rigorous" answer to some of your unanswered questions :) And if you spotted some errors, please correct!
  • 11 replies
    • Ejeckt
      Ejeckt
      Bronze
      Joined: 06.12.2008 Posts: 517
      Ok, so what you're saying is that the more I play, the worse my winrate is gonna get?

      So depressing...


      :s_biggrin: Kiddin. Nice post. I took a stats course in varsity, so I know what you mean in your post. Good work. Accurate and informative :)
    • OrcaAoc
      OrcaAoc
      Gold
      Joined: 06.08.2006 Posts: 4,813
      One factor that is important here is which assumtions you are making how good a player is before you look at those samples. In short it means if you have a strong indication that the player you are investigating might be a winning player relatively likely you need much less hands to 'proof' that.
      Mathematically this can be described with the Bayes' Theorem. You can find all this in the book 'The Mathematics of Poker' By Chen and Ankenman which I think you should absolutely get.
    • justkyle88
      justkyle88
      Bronze
      Joined: 07.05.2008 Posts: 7,596
      I don't think it mean your BB winrate will be neccisarily be smaller but there will be alot less varience.
      But who's to say that over that amount of hands a player can't improve or decrease is skill?
    • gavinonymous
      gavinonymous
      Bronze
      Joined: 18.12.2008 Posts: 1,146
      I tried using the variance simulator.. http://www.pokervariancesimulator.fr/ It seems to give excellent simulations of what your variance might look like over 100000 hands or more.

      I'm not sure what to put in the SD$ ($/100hands) box. Is it just the blue line from the normal PT3 graph in $/100, or should I filter to showdown hands and put that win rate ? I think more likely the first option.

      Also does anyone know what the SD0 line means?

      I've been doing some simulations on a smaller sample size and look at the %distribution of players. I seem to be in the 70%-80% range so I look at those results over many simulations.

      It's amazing some of the routes a random walk can take to end up at the same point over a long time. Some of the worst ones might have me give up poker altogether!
    • delete461
      delete461
      Bronze
      Joined: 04.07.2008 Posts: 1,036
      Originally posted by justkyle88
      But who's to say that over that amount of hands a player can't improve or decrease is skill?
      my thoughts exactly - over the course of 2 million hands needed to measure a winrate to within 2.2bb/hh, the player will gain a wealth of experience and probably improve considerably - especially if he is also studying the game. This means by the time you have played the 2 million hands needed to get the result, the result you get is wrong.
    • andyb43
      andyb43
      Bronze
      Joined: 23.07.2008 Posts: 903
      Is win rate the important thing? Is it worth getting hung upon?

      'Are you making enough money?' seems more relevant to me
    • AugustusCaesar
      AugustusCaesar
      Bronze
      Joined: 24.05.2008 Posts: 377
      Hi there,

      Very nice to see someone else also interested in a bit of theory (check my blog -link in the signature- if you like) :f_grin:

      I have 2 questions regarding the CI calculation:
      1) Where is the 80BB standard deviation from ? That differs very much game to game IMO, for example I would say that certainly player who plays SH PLO would have more variance than a player who plays NLHE fullring, as well as a DoN player would have less variance than a MTT player
      2) In the calculation you have :
      100*(0.05 - 1.96*8/sqrt(n)) = 5 - 800 * 1,96 / sqrt(n) ,
      shouldnt that be (the same for the right side of the interval with + ):
      100*(0.05 - 1.96*0,8/sqrt(n)) = 5 - 80*1,96/sqrt(n)

      PS. Once again, great to have someone else interested in a bit of a theory, if you are about to write a blog, I would certainly be reading :s_grin:
    • AugustusCaesar
      AugustusCaesar
      Bronze
      Joined: 24.05.2008 Posts: 377
      Originally posted by delete461
      Originally posted by justkyle88
      But who's to say that over that amount of hands a player can't improve or decrease is skill?
      my thoughts exactly - over the course of 2 million hands needed to measure a winrate to within 2.2bb/hh, the player will gain a wealth of experience and probably improve considerably - especially if he is also studying the game. This means by the time you have played the 2 million hands needed to get the result, the result you get is wrong.
      Actually I think that is the point .. Measuring a winrate is very unprecise way to see your results in poker, that was the point of the article I believe, as there can be 2 players with exactly same level of skill and the difference between their winrates can be huge ..
    • cindy1985
      cindy1985
      Bronze
      Joined: 13.11.2007 Posts: 358
      Originally posted by AugustusCaesar

      I have 2 questions regarding the CI calculation:
      1) Where is the 80BB standard deviation from ? That differs very much game to game IMO, for example I would say that certainly player who plays SH PLO would have more variance than a player who plays NLHE fullring, as well as a DoN player would have less variance than a MTT player
      2) In the calculation you have :
      100*(0.05 - 1.96*8/sqrt(n)) = 5 - 800 * 1,96 / sqrt(n) ,
      shouldnt that be (the same for the right side of the interval with + ):
      100*(0.05 - 1.96*0,8/sqrt(n)) = 5 - 80*1,96/sqrt(n)


      I will check your blog, in the near future, (not immediately since I am planning my move from China to France... )

      Thanks for your comment. Indeed the point of this post was to show that you can't infer your winrate in less than 2.10^6 hands, therefore we shouldn't focus too much on this kind of measure to see "how good" we are. Focus more on learning in forums, etc., and don't get overconfident when we have a huge winrate over a 100 000 hands period.

      Concerning your questions:

      1) The 80BB/100 hands is an average of some samples I have. These players are all Fullring NLHE players.
      I mention a very solid player, the only one I saw having such a low variance (40BB)

      I suspect in SH the variance to be a little higher... and even higher in other games.

      2) If the variance is 80BB for 100 hands, it is 80/sqrt(100) for one hand...


      Concerning the comment on the assumption I made, it is true that they are too simplistic, and I will definitely try to look at this book. It is also possible to perform the statistical test using the assumption that we have a winrate larger than a given value. It doesn't improve much the lower bound of our (positive) winrate estimation...


      On a side note, "Are you making enough money" is an even worse measure than the winrate (and they are linked anyway).

      Can't access the http://www.pokervariancesimulator.fr/ website, half of the internet is not working these days in China due to a submarine cable that has been damaged...

      Thanks all for your comments.
    • rjacobs003
      rjacobs003
      Bronze
      Joined: 07.01.2008 Posts: 45
      Sample size aside, where are you getting SD/100 from? I know PT2 had a stat but wasn't it meaningless in that it measured SD per sesison and not per hand? Also, what happens to the TAG players 80% hands folded preflop? Surely that distorts a variance estimate?
    • Kruppe
      Kruppe
      Black
      Joined: 20.02.2008 Posts: 2,144
      there's an SD/100 stat in HEM i believe