Since 28th June 2012, you are supposed to pay your social security in the country where you are based. Therefore this is important to be able to define your base.
EUR-Lex - 32012R0465 - EN - EUR-Lex
Then, you should look at your tax residence. It is important to note that in some case, you can be tax resident in France, but not have to pay tax there if your income is coming from another country which has a double taxation agreement and/or you have paid (enough) tax in this country. Therefore you should look at this agreement.
impots.gouv.fr has plenty of info, mostly in french, I do not know if there is an english version.
Being self employed can make it complex to arrange as you may have to deal with different countries for the tax aand social security...