Here's an algorithm:
Decide on the number of players per level of payout. Double elim would be 1-1-2-2-4-4-8-8... while single is 1-1-2-4-8-16...
Define the minimum fraction of players to be paid (F=1/6)
Define the ratio between payouts for adjacent levels (R=1.5)
Note the number of players (N) and the prize fund (P).
Take as many levels as required to pay enough players. For example with 1000 players and single elimination you would pay 1+1+2+4+...+128 = 256 players to pay at least 1/6th of them.
Starting with the lowest level, set their "share" to 1 each, and calculate the total number of shares to be paid (number of players at each level times R to the appropriate power, starting at 1 for the lowest level)
Divide the total prize by the number of shares to get the dollars per share.
Multiply the number of shares at each level (R to the Nth power) by the dollars per share to find the prize at each level.
The above will give non-round numbers. For round numbers, do the calculation, and then round down the amount for the first level (lowest level paid), and subtract their total prize from the prize fund. Remove them from consideration and do the calculation again with a reduced fund and number of players. Round down the lowest level again and repeat.
It is more or less trivial to program the above, and anyone familiar with Excel should be able to do it.
If you want to get fancy, you could have non-uniform ratios of payout between levels and put them into a table.