Currently Tally is only configured to accommodate two decimal places on invoices, receipts and bank transactions.
If you attempt to enter more than two decimal places, the data will round up. If you find the figure generated is incorrect, the only work around is to adjust/edit the invoice manually. To do this, set the quantity to 1 and enter in the total amount.
Also, please note that it is not currently possible to reformat the decimal separator to anything other than a period, for example, "€1.000,50".