HU Postflop GTO software (JBMCCFR)

    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      I'm starting this thread so that I can refer to it when people ask me what I'm doing over in my blog.


      I've given up playing much poker since July to work on a software idea that I had. The key points are:

      1) Although HU poker must have a GTO solution, the same cannot be said about multiplayer games. I wrote an article about this:

      http://www.modernpokereducation.com/preview/bonus_maths_5.pdf

      2) If two players get HU to the flop, from that point onwards the game is two player and zero sum, and therefore must have a GTO solution. Specifically, for a given board and given ranges on the flop, we have a two player zero sum game that must have a GTO solution.

      3) I have read the literature on solving games like this, and implemented the current method of choice, Monte Carlo Counter Factual Regret minimisation (MCCFR) in MATLAB, with a mex file to some C++ that does the main calculation. The bet sizes are restricted to a finite number of possible fractions of the pot at each decision point. I have implemented my own lossless bucketing algorithm. This algorithm is guaranteed to converge to the exact solution, with no loss of information.
      EDIT: Actual, some information is lost.

      I have just started making solutions available, along with some software to investigate it, at

      http://tinyurl.com/JBMCCFR

      Be sure to tick 'Create a Desktop Shortcut' during installation, because there seems to be an issue with running it from the command line.

      EDIT: Latest version of the interface is smaller, to fit on a laptop screen.

      -------------------------------------------

      At the moment I'm working my way through the 10 example hands at the end of Applications of No Limit Holdem, currently computing a GTO solution to Hand 2. I'm not yet sure exactly how many iterations I need to get a good solution but, even though there's some indication that a couple of thousand is OK, I'm doing 10K sweeps through the game tree to be sure, which takes about 3 days on my PC. This software is not a quick fix.


      My main objectives over the next month or so are:

      1) Improve the efficiency of my data storage so that I can compute solutions on larger game trees (more bet sizes, wider ranges).
      2) Read the literature on NLHE to work out how to sample the tree efficiently with a wider range of bet sizes. (Average Strategy Sampling looks interesting)
      3) Improve the efficiency of the code as far as possible.


      Note that my qualifications to do this are (i) I'm a mathematician, (ii) I love poker, but sadly NEITHER (iii) I'm an expert at computational game theory NOR (iv) I am a good poker player. Please bear this in mind when interrogating me!

      EDIT: Significant progress since I first posted this. See later posts.
  • 40 replies
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      I've just posted an update which fixes a few bugs and adds totals for each action on each street. It also cleans up the solution when a player is calling all in, so that he calls with the calculated percentage of hands, but with the strongest hands. This is a common problem when calculating what hands to call with in a GTO solution.
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Apparently the latest version didn't upload from my laptop last night, and my laptop's at home. :facepalm: I'll have to upload it when I get home. :f_mad: In the meantime, I've uploaded the data for Janda hand 2 (7K iterations looked like it was plenty).
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Originally posted by jbpatzer
      Apparently the latest version didn't upload from my laptop last night, and my laptop's at home. :facepalm: I'll have to upload it when I get home. :f_mad: In the meantime, I've uploaded the data for Janda hand 2 (7K iterations looked like it was plenty).
      OK. Latest version of JBMCCFR posted, but I've moved the files. You can now find them at

      tinyurl.com/JBMCCFR
    • SHAKA1992
      SHAKA1992
      Silver
      Joined: 10.07.2010 Posts: 90
      Originally posted by jbpatzer
      Originally posted by jbpatzer
      Apparently the latest version didn't upload from my laptop last night, and my laptop's at home. :facepalm: I'll have to upload it when I get home. :f_mad: In the meantime, I've uploaded the data for Janda hand 2 (7K iterations looked like it was plenty).
      OK. Latest version of JBMCCFR posted, but I've moved the files. You can now find them at

      tinyurl.com/JBMCCFR
      i need download matlab for running your algorithm??
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Originally posted by SHAKA1992
      Originally posted by jbpatzer
      Originally posted by jbpatzer
      Apparently the latest version didn't upload from my laptop last night, and my laptop's at home. :facepalm: I'll have to upload it when I get home. :f_mad: In the meantime, I've uploaded the data for Janda hand 2 (7K iterations looked like it was plenty).
      OK. Latest version of JBMCCFR posted, but I've moved the files. You can now find them at

      tinyurl.com/JBMCCFR
      i need download matlab for running your algorithm??
      No. If you run the installer, it automatically downloads the MATLAB Runtime Environment (a chunky 450MB) as part of the installation. You can then run the solution viewer as an executable. Remember, I'm not releasing the code that generates the solutions (yet), just a solution viewer.
    • SHAKA1992
      SHAKA1992
      Silver
      Joined: 10.07.2010 Posts: 90
      when imn running JBMCCFR.exe, the program show this error:

      ?throw_exception_v4@internal@tbb@@YAXW4exception_id@12@@Z in the tbb.dll
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Originally posted by SHAKA1992
      when imn running JBMCCFR.exe, the program show this error:

      ?throw_exception_v4@internal@tbb@@YAXW4exception_id@12@@Z in the tbb.dll
      God knows what that is. I compiled the executable to run on a 64 bit Windows 7 machine. Is that what you're using? I'm no expert on this stuff.
    • VorpalF2F
      VorpalF2F
      Super Moderator
      Super Moderator
      Joined: 02.09.2010 Posts: 8,901
      Originally posted by SHAKA1992
      when imn running JBMCCFR.exe, the program show this error:

      ?throw_exception_v4@internal@tbb@@YAXW4exception_id@12@@Z in the tbb.dll
      What OS and version?
      All patches up to date?

      Can you find tbb.dll on your hard drive?
      If so, post its full path, date and size.

      to make it faster, do you have any OTHER than:
      C:\Program Files (x86)\Intel\OpenCL SDK\3.0\bin\x64\tbb\tbb.dll
      2013-04-23 05:23
      195K

      C:\Program Files (x86)\Intel\OpenCL SDK\3.0\bin\x86\tbb\tbb.dll
      2013-04-23 05:25
      151K

      If you do, it is *possible* the software is finding the wrong one.

      Cheers,
      --VS
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Just added the data file for Janda Example Hand 3.

      I will have an update this weekend that adds the hot and cold equity and the expected value of each player on the flop, and also a hand by hand breakdown. I've also changed the layout a little and added what % of each hand bucket has made it through to a given street. I think it'll make the interface a lot more usable.
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Version 1.4 now uploaded to tinyurl.com/JBMCCFR. I think there's all the information that anybody could require there in the gui now. Apart from bug fixes, I'll probably not be updating it for a while as I'll be working on increasing the range of betsizes in the game tree.

      I've started computing the solution for Janda example hand 4, which should be done by early next week.

      Looking for some feedback please.

      Thanks,

      JB
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Solution for Janda Example Hand 4 uploaded.
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Solution for Janda Hand 5 uploaded.

      A quick update.

      I've not been able to use the data structure that I thought I would for some tedious technical reasons, but I do now have a structure implemented that will allow me to work with bigger game trees. I also updated the river hand bucketing algorithm, which I realized to could be more aggressive without losing information. This may speed up convergence a little.

      My plan is to redo one of the 3bet pot examples to check that my new code gives the same solution. I'm then going back to Janda Example Hand 1 and redoing it with 0.5x, 0.75x and 1x pot bets allowed on all streets. I think that will converge in a reasonable time. I also want to work out how to parallellize the MCCFR algorithm with the C++ part of the code. Apparently I'm going to need to use Boost.Thread and learn about multithreading. Fun, fun, fun. :f_biggrin:
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      I've posted the GTO solution of Janda Hand 1 with 50%, 75% and 100% pot raise sizes allowed, and also the solution to GTO Brainteaser #8 from the GTORangeBuilder blog at tinyurl.com/JBMCCFR. These use a new data storage format, so you'll need to update the software to version 1.5, an installer for which I've also added. The new version can still read the old data format, so it should work with any older data files you've copied. It has also been updated to allow solutions to start from turn or river, and I've made it a bit bigger because there's so much information in the multiple bet size solutions (e.g. fold, call, raise 0.5 pot, raise 0.75 pot, raise pot, or jam). Any ideas on the best way to display the data appreciated.

      Now, multithreading........
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Originally posted by jbpatzer
      I've posted the GTO solution of Janda Hand 1 with 50%, 75% and 100% pot raise sizes allowed, and also the solution to GTO Brainteaser #8 from the GTORangeBuilder blog at tinyurl.com/JBMCCFR. These use a new data storage format, so you'll need to update the software to version 1.5, an installer for which I've also added. The new version can still read the old data format, so it should work with any older data files you've copied. It has also been updated to allow solutions to start from turn or river, and I've made it a bit bigger because there's so much information in the multiple bet size solutions (e.g. fold, call, raise 0.5 pot, raise 0.75 pot, raise pot, or jam). Any ideas on the best way to display the data appreciated.

      Now, multithreading........
      Oooooooo! I think I've found a bug in my indexing routine for the new Janda hand 1 data file. I'm going to have to recompute it. Treat with a pinch of salt! :facepalm:
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Originally posted by jbpatzer
      Originally posted by jbpatzer
      I've posted the GTO solution of Janda Hand 1 with 50%, 75% and 100% pot raise sizes allowed, and also the solution to GTO Brainteaser #8 from the GTORangeBuilder blog at tinyurl.com/JBMCCFR. These use a new data storage format, so you'll need to update the software to version 1.5, an installer for which I've also added. The new version can still read the old data format, so it should work with any older data files you've copied. It has also been updated to allow solutions to start from turn or river, and I've made it a bit bigger because there's so much information in the multiple bet size solutions (e.g. fold, call, raise 0.5 pot, raise 0.75 pot, raise pot, or jam). Any ideas on the best way to display the data appreciated.

      Now, multithreading........
      Oooooooo! I think I've found a bug in my indexing routine for the new Janda hand 1 data file. I'm going to have to recompute it. Treat with a pinch of salt! :facepalm:
      New version uploaded.

      I note that, when this file is loaded, it takes up 1.7GB of RAM, so beware if you don't have much RAM. This is clearly something I'm going to have to look into if I want to distribute larger solutions. The file is only 0.28GB, so it's clearly reasonably well compressed. I can already halve the RAM requirement by saving it as single rather than double precision numbers, but I'll have to work a bit harder than that if this is going to be practical.
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      I added the file

      Ad7s5d_BB3betvUTGcall-Janda_manysizes_gui.mat

      which allows for 0.25, 0.5, 0.75, 1, 1.25 and 1.5 pot bets. I think the solution is quite interesting, as there are several spots where 0.25 pot bets are in the solution.

      I can see that the last gui I released is a bit big for laptop screens, so I will release a smaller version some time soon.
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      After discussing hand bucketting with Alex Sutherland, I think I've done it wrong in the sense that I've lost some information. This doesn't mean that the results I have aren't useful, but there is some information loss in my bucketting algorithm. I basically haven't taken into account range v range card removal effects. I'm going to redo the bucketting algorithm and see what effect it has. Sadly, it wil for sure make the code slower. :f_cry:

      That's beta testing for you.... :facepalm:
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      Originally posted by jbpatzer
      After discussing hand bucketting with Alex Sutherland, I think I've done it wrong in the sense that I've lost some information. This doesn't mean that the results I have aren't useful, but there is some information loss in my bucketting algorithm. I basically haven't taken into account range v range card removal effects. I'm going to redo the bucketting algorithm and see what effect it has. Sadly, it wil for sure make the code slower. :f_cry:

      That's beta testing for you.... :facepalm:
      After further discussions, here's my thoughts and plan.

      1) thoughts posted here Midlife Crisis? What Midlife Crisis?

      2) Test this by calculating an equilibrium for a pair of ranges with maybe 40-50 combos, firstly with my bucketting and secondly without to see whether there's any significant change.

      3) Calculate the epsilon distance of my abstracted solution within the unabstracted game. This is a measure of how close my solution is to the Nash equilibrium (epsilon = 0).

      4) Drink lots of wine.

      I think that's a good plan, especially the last bit. :f_biggrin:
    • jbpatzer
      jbpatzer
      Bronze
      Joined: 22.11.2009 Posts: 6,944
      I have implemented by new bucketting algorithm, and so far it seems to work, although I need to do some more testing. Unfortunately, for Janda hand 1, it increases the number of buckets by a factor of just over two, and the computation time by a factor of about six. This is not good. I will focus on this example for testing and work whether there's much difference in EV or equilibrium strategy between my old and new bucketting algorithms. I'll work out the exploitability of each as well - something I should have done ages ago really. That's all probably going to take while, so don't hold your breath.

      I've added an installer for a smaller interface version that should make everything visible on a smaller laptop screen. If anyone still can't see the flop, let me know.

      When installing the software, make sure you check 'create a desktop shortcut', or it may not work.