You can take the theoretical exams in any EU State however; before doing so you will need the recommendation of a Head of Training of an ATO. The HT is also required to certify that you have completed 100 hours of theoretical training. As there are probably no ATOs approved yet for PPL training under EASA, it may take a while for the system to develop fully, but that should not stop you doing the exams where you want. Just bear in mind the theoretical training may differ considerably from one State to another as there are no standardised question banks.
The licence can only be issued by the State that holds your medical records.