The problem with patterns is that the underlying reasoning could literally be anything, and if there is no context to the problem then all answers are basically correct. For example, what's next: 1, 2, 4, 8, 16, __? Did you think it's 32? No, it's 31. This pattern represents the maximum number of sections a circle can be divided into if you connect N dots on the circumference, starting from one.
My answer to what is decimal 10 in this system is:
First of all note that the problem states that this is a new number system only, so all the algebraic rules should still hold. Using this information we can deduce that:
1) 2 + 1 = 2 then 1 is 0 since anything plus 0 will just be itself.
2) 3 + 3 = 4 and 4 + 3 is 6 in decimal; 4 + 3 = 3 + 3 + 3 so 3 has to be 2 in decimal, and 4 will be 4 in decimal.
3) 2 + 2 = 3 and since 3 is 2 in decimal then 2 is 1 in decimal.
4) 4 * 3 = 5 and 6 = 5 / 3 so 5 = 4 * 3 = 6 * 3 from this we know that 4 = 6 in this system, so 6 is also 4 in decimal.
5) 5 = 4 * 3 = 4 in decimal * 2 in decimal = 8 in decimal
Now that we have all the given numbers assigned to their correct counterparts we see something strange happening:
The system, as converted to decimal increases from 1 to 5 but then goes back, so we have this pattern: 0, 1, 2, 4, 8, 4.
This is where my opening paragraph takes hold, since number 6 went back to 4 for some reason, all bets are now off. Where will 7 be? Keep going back or start going forward or something else?
The most reasonable conclusion to the pattern would be 2, 1, 0 in my opinion. There is not really much else I can think of to do here. So 7 will be 2 in decimal, 8 will be 1 and 9 will be 0.
Now the question becomes what will happen after the pattern reaches 0 again. Will it go back to 2, 4, 8? That seems uninspired. Seeing as how this is a number system, all the numbers should theoretically be represented. I'm sure you've noticed that so far all we've had are powers of two. Taking a huge leap of logic here, I believe the pattern will now start with powers of 3, increase 3 numbers just like before and then decrease back to 0. So 10 will be 1 in decimal, 11 will be 3, 12 will be 9, 13 will be 27, 14 will be 9 and so on back to 0. After this the pattern will restart with powers of 4.
Having concluded what the number system does all that is left is to determine what 10 decimal will look like in this system:
since 10 is not a perfect power of any number other than 10, we will need to wait until this pattern reaches all the way to powers of 10 and that happens when the number is 67. So 10 decimal is 67 in this system...
Interesting to note the problem states that after the radix point the system is just regular base 10. So since 0.5 is thought to be five tenths or one half of something this means that 3.5 will be half-way between 3 and 4 in this system. That means that:
Since 4 is 4 in decimal and 5 is 8 in decimal then 4.5 will be 6 in decimal! This means that each decimal number can be represented by infinitely many fractions, going both forwards and backwards. Even without my assumptions of changing power bases, 4.5 should be 6 in decimal as it is halfway between 4 and 5, but so is 5.5, since halfway between 5 and 6 is going back. So 6 in decimal will be 4.5 and 5.5 in this system and others as well. This means that 10 in decimal can also be 12.055555 and 19.5 and 27.25 etc.